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H (X) I Renyl’s «<-eatropy associated with a 

® generalls®<i discrete probabili^ scheme. 

(X|Y) : =<-entropies associated with two 
^ univariate probability schemes 

H^(y/X), EXx/Y) specified by the randcai variables X 

^ and y or equivalently “(-entropies 

associated with a two-dimensional 
channel. 


I^CX ; Y) 


“(-information conveyed about X by Y or 
equivalently “C-information of type I. 


Icc(Y 5 X) 

( 1 ) 

Ccc 

( 2 ) 

0. 


i “(-information conveyed about Y by X or 
equivalently “(-information of type II. 

: “(-capacity of type I . 

* “(-capacity of type II. 


I^<X; Y3^,Y2,...,Y^) 


“(-information conveyed about X when an 
input is received by m outputs. 


Ui Y^s Yg) 


“(-information for three alphabets 

X| end Yg. 


Ihe symbols with the subscript “< replaced by i,e.g* 
Hj^(X), Ij^CXjY) etc., will represent the corresponding 
Shannon* s definitions* 

All other symbols used are understood front the text. 

Prefix *“(♦ stands for ’ ‘of order * , e. g* '“(-entropy* 
would mean entropy of order “(* 

Ihe unspecified base of the logs is assumed to be 2 
but wherever a change of base is needed for operational 
facilities in differentiation and integration, such a change 
has been indicated in the start e*g. from section 2 onwards 



of part I, claapter Sybase of the logs has been assumed 
to he e ait«i this change has been indicated at the end 
of section 1, part I, chapter 5. Very occasionally the 
base is even mentioned 4^st for the sake of emphasis* 
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A thesis suhraitted 

In partial fulfilment of the requirements 
for the BegTee of 
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by 

P. H. Ghhabra* M.A. (Math. ), M.A. (Math.Stat. ) 

to the 
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In the present thesis* taJting Benyi's '=<-et\tropy 
(entropy of order '=<, < > 0, «< 4* 1) discussed in paper 
’’On Measure* of Entropy and Information, Proc.4th Berkeley 
Symp, Math. Stat. Prob. , 196i, pp. 647-661*' as the basic 
measure of uncertainty, other ^^-entropies '=c^ 0, <=< 4* 1 
associated with a chantiel are defined and efforts have 
been made to generalise most of the basic concepts 
available in the present-day infoisaation theory. All the 
resi:iit|;s, except the two results on maximisation of !^(X) 
and the evaluations of °<-capacities based on one of th^, 
developed in the thesis are true for 0, 4“ 1 tend 

in the limit, as 1, to the corresponding available 
results. The of the thesis, except the section on 

'Generalised Entropies* in chapter 1 on Introduction, deals 
with ccaoplete distributions. 
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Chapter 1 introduces the concept of generalised 
entropies and gives the chap ter- cum-section-wise 
siimmary of other chapters. 

In Chapter-S, ®t.eatropies 0, «< 4* If 
associated with two discrete prohahility schemes specified 
by the random variables X and Y are defined as follows* 

H^(X) a (!/!-'<) log 2 p'^Cx. ) 

HXY) * (l/l-=() log 2 p'^Cy.) 

i^(X,Y) * (l/l-'O log 2 

ij 

^<(Y/X) as (1/1-°^) s p(x^) log 2 

1 J 

2 pirp log 2 P^ix^/yp 

iS i 

Sc^e of the properties of these “^entropies are 
discussed* Some basic inequalities connecting th^ and 
tending ‘in the limiti as *< -• 1, to Shannon’s basic 
results of the types 

EjiJ/X) S H^(Y) 

and H^(X,Y) * Hj^(X) + H^(T/X) 

are developed. jJ.! the results developed for two discrete 
probability schemes are extended for any finite number of 
such schemes* In the last section, alternative definitions 
of conditional «c-entropies 
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H CT/X) 3t (!/!-«() log 2 p(x. ) 2 p (y^/x. ) 

i 3 


\iJ/iy « (l/l-°() log 2 p(y^) 2 p <Xj^/yj) 


are developed aad the truth of vaid.ous results deve]h:>ped 
in the chapter is ascertaiiied In the light of these m\f 
definitions. 

In Chapter 3, two types of ^c-informationsj^tJ^ 0, 

“( ^ Intending in the limit* as 1* to the available 
Shannon* s concept of mutual information, are developed and 
based on them* the concepts of ®<-ca®>acities, oc-redundancies 
and «<-efflcimacies are defined. In the light of these 
concepts of order ^ the characteristics of noiseless channels, 
chgpnels with independent input and output, charnels with 
symmetric noise structure, BSC, BBC and cascaded channels 
are considered. oClnfoimation for a single input and 
several outputs is also defined and the available a<!^itive 
property of corresponding mutual information Is generalised 
i.e. proved valid for the concept of "C-^infoimatioa. «<• 
information for more than two alphabets is also defined. 

All these results are ascertained to be valid for alternative 
definition of %j(T/X) and ^(X/X). 

In Chapter 4 a generalisation* based on the 
concepts of «<*info 2 mation of type II and *=<-capaclty of 
type II, of Shannon’s second fundamental theorem for noiiy 
discrete memoryless chinnels is stated and an heuristic 
proof for BSC is given. AviilahL© Fsno bound for discrete 
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memopyless channels is generalised and as a consequence 
of this generalisation, a generalised form of the 
converse of Shannon* s second fundamental theorem is 
stated and the proof is given for a family of extensions 
of a discrete memoryless channel. A generalisation, based 
on of Shannon’s theorem for noiseless coding is 

al^ made. Bounds for ^ ^ ^^^i^ “i ^ terms of H^(X) 

are developed and two results, one dealing with shannon** 

Fano encoding and the other known result on the attainment 
of L its lower bomd, are derived as corollaries of these 
°(-tiounds. Some probl^s that arise frcai these bounds are 
also stat^. All the resiilts develop^ in the chapter are 
ascertained to be valid for alteinative definitions of 
conditional «<“entroples. 

In Chapter 5 the notion of ifenyl’s “^-entropy is 
extended for continuous and multidimensional contlniaous 
schemes. Based on these extensions, other “^-entropies 
associated with a bivariate a>ntinuous and a multidimensional 
continuous channel are defined and the various results 
developed in Chg^ter 2 for discrete channels are extended 
for continuous and multidimensional continuous channels. 

Some mathematical drawbacks in adopting these extensions as 
measures of mcertainty are brought out and the concepts of 
«<*-infoimations, tending in the limit, as 1, to the 
available concept of the rate of InfoxmatLon processed in 
such channel), are defined. It has been shown that these 
concepts of “t-infopmations don’t have the salient drawbacks 

-I r» 
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In Chapter 6 '=<-capacltles for multidimetisional 
continuous chap.nels are defined and as a need for their 
evaluations the problem of maximisation of ;^(X) is 
tackled for the following three types of contratnts on X: 

(1) when the domain of X is bounded i»e. 

^1» 

(li) when the components of X assume only non- 
negative values and have specified first moments. 

(ill) when the ccmponents of X have specified 
standard deviatlonst 

^^-Information*;, when both the input X load output X are noimally 
distributed| are evaluated, “^capacities are also evaluated 
when the chmanel is subject to 

(t) additive CJaussian noise and each component 
of the input vector is distrifcuted with mean zero and a 
specified standard deviation. 

(ii) besides conditions in (i), input is also 

Gaussian. 

A generalisation of Shannon-Hartley channel-capacity formula 
is also made. 

In Ciiapter 7 «C-entroples associated with a multi- 
dimensional semi-con tinuous m^oryless channel are defined, 
available Fano bound for such channels Is generalised and 
as a cK>nsequenee of this generalisation, a generalised foxm 
of tite converse of Shannon’ s second theorem for semi -continuous 
mmoryless channels is stated and its proof for a family of 
extension of such a channel is outlined. 
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In part I of Chapter 8, «c-entropies associated 
with different steps of a stationary Harlsov chain, when its 
Initial state prohahllities are specified, are defined 
and two theorems connecting these «<-entropies are given. 
Sc«ne of the results of ttiese theorem* tend in the limit, 
as oc 1, to the known result on the additivity of 
corresponding Shannon’s entropies. 

In part II of Chapter 8, =<-entropy associated 
with a stationary source is obtained and its existence is 
established when the source is identified with a stationary 
symmetric channel. The concepts of the rates of <=c-infoima* 
tions for stationary synanetric channels are also developed. 
These rates tend in the limit, as 1, to the known 
rate of information procesced in such channels. 
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IHIBDDPCTION 

1* MiaOPIES 

Shajmon (19^) gave for a complete discrete 
probability sch^e 


X « 


p{x^) 


P.i\) > 0 ;> 


i 


the expression 

H^CX) = - :| pCx^) log p(Xj^) 


n 

1 to n,2 p(x )=i 
i=l . ^ 

*..( 1 ) 


( 2 ) 


as a measure of entropy, Tbe quantity H^(X) has been chara- 
cterised by various sets of postulates j a weaXer set given 
by Fadeev (1957) is as follows i 

(!) H^CX) is a symiaetric function of p(Xj^) 

(ii) H^(p(x), l-p(x) ) is a continuous function of 
p(x) for 0 ^ p<x) ^ 1 

(iii) H^( 1/2 , 1/2 ) a 1 

Civ) H^C t p(Xj^), Cl-t)p(x^), p(x 2 ),...,p(Xj^) ) 

* H 3 ^(p(x^),....,p(x^))-i- p(x^) H^(t,l-t) 

for 0 ^ t 1 

!£hese postulates characterise E^(X) uniquely, fhe weakest 
set of postulates known at this time may be found in Lee 
(1964). Xapur (19®? b) has also made a study of these postulates, 



Tlue fact that <2ah he regarded as the mean 

Value of the variable logg Cl/p(x) ) was earlier emphasi- 
zed by some authors eiq?ecially by Bernard Cl 9 Sl). So 
keeping (2) in view» Renyi (3961) defined the concept of 
a generalised discrete probability scheme 

X « C ^ )i p(x^)> 0,1 a 1 to r,0< p(x. )< 1 

p(x2^)...,p(Xy) ial ^ 

... ( 3 ) 


and defined the entropy attached to it as 

r 

-2 P(x. )log p(x.) 
i»l ^ ^ 




-.-rrrnmr-n,**' 

S p(x.) 
i*l 


( 4 ) 


Renyi characterised H CX) by the following five postulates* 

X 9 

1 * %(X)g is a symmetric function of P( 3 ^)* 


2 , If 5 ^p<x)] denotes the generalised probability 
distribution consisting of the single probability p(x)y then 

^p(x)J )g is a continuous function of p(x) for 0<^(x).^ 1 

3 . i\l/ 2 ] )g * 1 


The other two postulates need the introduction of scase 
notations* let A denote the set of all finite discrete 
generalised probability distributions associated with genersr 
Used discrete schemes. For PU), P(T)£ A , let P(X) ♦ PCY) 
denote the direct product of PCX) and P(t) i.e* the distribu- 
tion consisting of the numbers p(Xj|^)p(yj) with 1 *1, ..,r , 
i » 1,,.,, s and if I p(xj^>i-| pCy^^ 1 then let 
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PCX)UP(y) * j • • * • jP( 2^) jp(y^) j * » * , 

M®te that If 2 pCx^) * 2 p(yj) 1 than P(X)up(Y) 
is not defined* So the last two postulates are 

4* E^iX * y)g « %Cx)g 

Cz^pCxj^)) \{x)^+i ^p(yj)>%<'g)g 


6. I,(XUT)^ *- 

X M 


2 p(2^) + I p(y^) 


Further Henyi argued that if| in place of ordinary 
mean as Used for getting (2) and laid down in postulate 5, 
we consider the Kolmogorov- Kagmo generalised mean of the 
raadom variable lo| ^-- ^ '-j w.r.t. the strictly monotonlc 
and continuous parametric function 


f^(i) 




<y 0, oc 1 


or in other words if we replace the postulate 5 by postulate 
s‘ s 




J p(*i) ? pCJj) 


we shall arrive at the concept of entropy of order <• For 
the generalised distribution given by (3), it is given by 


r o( 

2 p (at^) 


i^(X) 


i«l 


g 


l-«< 


^ P(3Ei) 
i»l 


•< > 0| < + 1 


(5) 



( 6 ) 


In particular when ^ becomes 


H -CX) * 1« 

«C l-o( 


gg s P ) «< > 0, ^ 1 


It may be noted that both and \iX) satisfy all the 

/ 

postulates l>2)3y4 and d . 

?aBBa (I9d5» 1966a) gave two generallsationg of (6) 
hj defining 


(X) 


1 £ 


A 


^ ^ n - «< 


leg 




(n-l)< «<<n ,n ^ 1 


n ^ 

H CX) 

S n* ®( 

1 


lo 


g 


«</& 

f P PC3C^) 
S p(3l) 


0 <oc <n, n;^i 


and studied their properties like mono ton! city ©to. It may 
be noted that these entropies are two parameter entropies* 
Later on Kapur <19€7a) generalised (S) by defining genera * 
Used entropy ©f order ®< and type p s 


CX) 


S 1* «< 


log 


*<#■ P-1 

? » <*i) 

JL 


2 PCx^) 


> 1 ,<>0, eC 1 

( 7 ) 


and studied its properties* It may be noted that in the 
limit as «(-• 1 and p * 1 (7) tends to (4) . 
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In the present tlaesis testing (6) as t3m basic 

of 

measureientxopy) other entropies of or^er «< associated 
with a channel are defined and efforts have been made 
to generalise most of the concepts available in the 
present day infoimation theory* It is possible that 
almost all the concepts involving Sh^ganon's measture of 
infoimation in the present day information theory may be 
generalised to give •’Information Theory of Order ej* ' in 
its full form* In that case the present available theory 
would be just a limiting case, as $ of the generalised 

••Information Theory Of Order «(** • However the difficult 
ties in developing this generalised theory in its full form 
are considerable* 

The following is the chap ter- wise summary of what 
is contained In this thesiss 

Chapter - 2 

In section 1 entropies associated with two 
discrete schemes are developed. 

In sections 2.1 and 2.2 it is proved that these 
entropies are continuous and monotonically decreasing 
fmctlons of ^ and in section 2.3 their maximaHty is 
discussed. In sections 2.4 and 2.5 an easier method for 
finding successive derivatives and hounds for second 
derivatives of these entropies are developed* 



Im section 3*3 Sliannon's most basic and famous 
inequality I^CIT/X) < is generalised i.e. proved 

foir Corresponding <<*entropies« In section 3.3 t-wo 
infinite sets of inequalities^ each m^ber of either set 
tending in the limit) b.s Ij to the basic Shannon* s 
equality + H^(X/X), and two bounds, one 

for 0 < «C < 1 and the other for *<> 1, for X) 

are developed# In section 3.4 Khinchln’s generalisation 
of Shannon’s inequality H^(X/X) < H^(X) is generalised. 

In section 3.x and 3.2 two more relations betwem these 
•c-entrc^ies are developed. In section 3.6 a study of the 
validity of i^<(X,X) « H^(X) * ^ (X/X) for «< > 0 is 
made. 

In section 4 various main results developed in 
section 3 are generalised for r discrete schemes. 

In section 6 alternative definitions of conditional 
*<«entropy and ^(-eqiai vocation are glv®a and the validity of 
various results developed in this chapter is explored In 
the light of these alternative definitions. 

Chapter - 3 

In section I <<»en trebles, associated with two schemes 
and defined in section i of chapter 2, are interpreted as 
those associated with a discrete memory less channel. 

In section 2 two types of ^c-lnfoxmations and hence 
two t^pes of capacities, redundancies g(. efficien- 
cies are defined. Definition of two types of «K-infoxmatioms 
are the results of the invalidity in general of 



^(X,T) = ♦^(T/X) » ^(I) + « > 0, "( 4 1 

la seetloa 3 '<**informations» when corresponding to 
a single input a nmher of outputs are receivedi are defined 
and the available result 

* * Inf oxiaation eonvejred about an ii^ut by a ©onpound 
observation does not depend on \ihether we consider the 
eoffipoui^ observation as a whole or broken into Its conponents' 
is generalised* 

In section 4 following McGill <1964) «C*information 
of several alphabets is defined and e^qpressed in terms of 
a(- entropies. 

In section 6, the available result on the leakage of 
information throuj^ a casoa^ of channels is generalised* 

In section 6, characteristics of noiseless channels, 
channels with independent input and oul^jut, channels with 
symmetric noise structure, BSC and BBC are considered in the 
light of «£• entropies and the concepts defined in section 2. 
Lastly in section 7 all these results are verified in the 
light of alternative definitions of conditional «<»entr©py and 
equivocation developed in section 6 of ©hap ter 2. 


Chapter - 4 

In seotion 2 a generalisation of Shannon’s second 
fundamental theorem for noisy discrete memoryless channel^ 
is stated by taking the infomatlon processed in the channel, 
as given by «<-infonttation ©f type II and a heuristic proof 
is given for BSC. 
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In sectLojci 3 tiie avadlali^Le Fano bound of discrete 
channels Is generalised and, as a eonsequeaoe of tM.s 
generalisation, a generalised foxm of tbe conTerse of 
Shannon* s second fundgaental ^eor^ is stated and proved 
for a family of ^tensions of a discrete maaoryless channels* 
In Abramson (1963) is given a generalisation of 
ShiAnon*s first fundam^tal theor<^. According to this 
generalisation every received symbol makes ns consider a 
D-ary code so as to find what has hem commnoicated* fhe 
proof of this is based on the concept of equivocation. 
Cspphell (3965) gave a generalisation of Shannon's first 
theorem for noiseless coding by considering an exponential 
cost function and taking Kenyi's «c-entropy as the basic 
measure of mcertainty* Following the lines of the genera- 
lisation given in Abr mason (396$) and based on our concept 
of o(-equivocation we have generalised Campbell's theorem 
and hence Shannon's theorm. This is done in section 4 • 
la section S.i, taking the usual average length 
Xm* 2 pix^)nj|^ associated with the transmission of a sequence 
of encoded words of lengths given by'^^a^] and respective 
probabilities of transmission given by ^p(^)J , generali- 
sations of the result L > SllS, are made. In section 6.2 

log D _ 

on ™ basis of these generalised bounds of 1, two resultSf 
one stated page 140, |©za (3961), on the Shannon-Fmio 
encoding procedure and the other^ a knom result dealing with 
the necessary and sufficient condition on the attainment of ^ 
Its lover bound, when the <K»rresponding entropy is measured 



lof Siattnxion*s defisltioiaii are discussed. Lastly In sectLon 5.3 
some probX^s that result from tliese bounds are given aM a 
note is added as to the validity of all these results for the 
alternative definitions of conditional ‘^•ontropy and equivo- 
cation. 

Chapter -6 
Part I 

In section 1 Benyi's °c-entropy is ^tended for a 
continuous scheme and some of its requisite properties are 
stated. 

In section 2 a(*^tropies asso(^ated vitb a tvo-dlmen- 
sional continuous memoryless cbaunel are defined. 

In section 3 some relations between these °(-eatropies 
are developed. These relations run parallel to the correspon- 
ding relations developed for discrete charmels in chapter 2. 

A salient feature of these relations is the validity of 

JS^ U,l) « ♦ H^CY/X) * ^(Y) +^(X/I) «{> 0 

when X and Y are Jointly Gaussian. 

In section 4 it is shown that for a continuous 

scheme) in ccsmparison to l^(x) for a discrete scheme) is not 
always positive) finite and Invariant even under linear trans- 
fozBations* 

In section B as in the discrete case two types of <>(- 
infozmatLons are defined m^d sm& of Its properties are 
considered* 
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In S 0 ctioii 6 «Co 6«paelties ar® deigned, flam proo®* 
dure of defiling <3C«cai>acitles for continuous channels runs 
parallel to that of discrete channels* 

In section 7 all the results developed in this Part 
are considered in the light of alternative definitions of 
conditional ‘<-ent3?opy and «c-equi vocation. 

In section 1 Benyi's “C-entropy is extended to a multi* 
dimensional continuous scheme. 

In section 2 «<*entropies associated with a multidimen* 
sional continuous memoryless channel are defined and seme basic 
relations between them are developed. These relations run 
parallel to the corresponding relations developed in part I of 
this ohapter. 

In section 3 all the relations developed in section 2 
are generalised for r multidimensional continuous schemes. 

In section 4^ as for two-dimensional channels) two 
types of °c*:ki formations are defined and some of their properties 
are proved. Lastly in seetion 5 all these results are considered 
in the light of alternative den.nitions of conditional <ac*entropy 
and «c-equt vocation. 

Chapter -6 

In seetion 2 ‘<*capacities are defined. 

In section 3 the problem of maxhaisation of ]^(X) is 
considered for three sets of constraints on X . 
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Xb sectloB 4 as a oonsequenoe of the result 
obtalBet by* maxlmisixLg B^(X) vhea the oompomeuts of Z 
have specified stmdard deviationsy the luvalldll^ isi 
general of 

^^(i) ^ «X > 0 

Is sbovB* 

In section 5.1 evaluations of «<-info»tBatLons^whea 
X and J are Oraussian, are made. 

In section 6.2 «c-»capacities are evaluated for tvo 
casest (i) the channel is effected by additive daussian 

noise and each ecmponent of the input vector is distributed 
with mean zero and a specified standard deviation, (li) Besides 
conditions in (i) input is also daussian. 

In section 6 a generalisation of Shannon* Bartley 
Channel* Capacity formxila is given. 

In section 7 validity of various results Is considered 
for alternative definition of coMitlonal «<-entr@py and *<- 
equivocatdlon. 

Chapter -7 

In section l "C-entroples associated with a s«nl* 
continuous mitltidimensional mmaoryless channel are defined. 

In section 2 Fano*s inequality for such channels is 
generalised. 

In section 3y as a consequence of the gone rail satiom 
made in i^c^on 2) a generalisation of the converse of Shannon’s 
second fundamcKital theory for seml*oontinuous momoryless 



- 12 - 


claazmeXs Is stated aiad tt« Haas, on vbleli it can te prorod 
for a fanlly of extmslons of snon a channel » are iaadieated* 

In section 4 these results ax© iierlfied fox alternative 
definitions of conditional “C-entropy and oc-equi vocation* 


Chapter -8 

Earif-JL 

Jn section 1*2 "t^entropies associated idth different 
steps of a Markov chain are defined* 

In section 2 some relations between these “C-entroples 
are developed in two theorems* theorem 1 gives two infinite 
sets of inequalitiesi each member of either set tending in the 
limit as 1 to the well Isnown result 

Jr) „{ 1 > 

(r) 

where is the Shannon's entropy associated with an r step 

transition probability matrix of a chain, theorem 2 establishes 
the Validity of x idien the transition probability 

matrix of a Markov chain is like that of a channel with symmetric 
noise structure* 

Part II 

In section 1*1 <-entropy associated with a stationary 
source is defined and in section 1*2 its existence is established 
wh^ the underlying channel has symmetric noise structure. 

In section 2 existence of other ^c-entropies associated 


with a stationary symmetric channel is established and the 
concepts of <<*infomatlon8 for such channels are developed* 



3. NOTES 


1. If we assume Shannon’s convention that 0 = 0 
the definitions for a(-entropies given in chapters 2,3, and 4 
are also valid when some of the p(x^) or conditional proba- 
bilities are zero. So this convention has been assumed in 
the developments of these chapters. As regards other chapters 
this point has separately been dealt with. 

2. In all this work complete probability distribu- 
tions are considered. A probability matrix p(Xj^) is governed 
by the conditions 

0 4^ pCx^> ^ 1, S p(x^) = 1 

3. The meaning of Of written against a result, 

is that the result is true for adl values of '’Cf howsoever 
large , except for °t = oo for which its validity is not 
explored. 



Taking Renyi*s entxopy as the baste meastire of 
uncertainty, various ®t-entroptes associated with two discrete 
schemes are defined in section !• Some properties like 
continuity, monotontetty etc. are proved in section 2. Some 
basic relations between entropies are developed in section 3 
and in section 4 these are generalised for any finite number 
of discrete schaBcs. In section 5 the validity of all these 
results is considered in the light of another definition of 
conditional entropy and ‘^-equi vocation. 

The case, when the underlying schenes are independent, 
has been taken into account while applying Jensen’s and H5lder’s 
inequali ties. 


1 . 


Let there be two complete schemes 


........ n 

( ^ ^ ^ S pix.) « 1 

p(Xi)»p(x2),..,p(Xj^) i*l 


p(yi)»p<y2)f •*>pCym> ^ 

Let p(3C^»yj) ^e the probabilities of the joi^nt occurrence 
of the events and y^ . The set of events ^ly^CK i< n » 
1 < 3 <m ) represent Joint complete scheme. Let pCy^Aj^) 







occurs given the occurrence of the event 35 ^ of the scheme 
X so that 

y^) »= pCxj^) p(yj/xi) 

and let pCxjj^/y^) he the probability of tlae event of the 
scheme X occurs, glvaa the occurrence of the event y^ of 
the scheme Y so that 

pCXj^, Yj) « p(yj) P<3£i/yj) 

These tvo schemes oith cox^ltlonal probabilities are also 
complete. Bence there are five ‘^-'entropies attached to the 
schemes X and Y s* 

(i) Ibr the scheme X 

^<(X) = -1— 1«* f P°‘(3ti) ”‘> 0, « + 1 (1) 

®C i 

(ii) For the scheme Y 

a^(Y) s — i— log 2 p'^Cy*) «<> 0, «< ^ 1 (2) 

!• o< i ^ 

(ill) For the Joint sclimme 

1 «< 

w (x,Y) » log S p (Xi,y.) <>0, ^ 1 (3) 

i,j '* 

(Iv) The conditional “(•entropy Et((Y/3^), calculatod 
on the assumption that the evmt 3 ^ of the scheme X has 
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©cciured is 

Bt<(Y/3c^) * log 5 p^Cy/x^) <<;> o, ^ 1 (4) 

To have ]^(I/x)» the ot-entropy attached to the conditional 
sch^e Y/Kf it is Impropriate to ts^e the expected valtte of 
(4> for all x^ } so 

(YA) * - j ' ” I p(aCi)log S p^'Cy^/xj^) (6) 

«<> 0,o( ^ 1 

(v) Similarly ^CXA>, “C-entropy attached to the 
conditional scheiae XA, is given hy 

^(XA) - I p<yp log 2 (6) 

«< > 0,oC fr 1 

It can he easily verified that in the limit, as 
^ 1> Cl) I C2)| C3)f CS) and C6) tend to the corresponding 
Shannon's deflni1d.ons . 

In the limit, as «<•»*» , tends to 

•log max p(x^ ) since 

Jl 

tt L(X) » It _i_ i«* S ^U. ) 

u 1/(1-«C) 

* log It ( Z p^Cx.) ) (7) 

0(^00 ^ i 

If pCx^) is ttie largest or one of the largest and since 
® PC3«^) ^ 1 , we shall have for 1 



oC 

So from (8) as , CS p ) tends to P(x^)« 

Using this t (7) reduces to 

It “ • iog max P(3s^) 

i 

•» 00 

Further since the limit of a weighted mean 
involving a finite number of terms is the mean of the 
limits of its individual terms, so we shall have 

ht ^(1/X) * • S pCXj ) leg max p(y-/x-> 

o( ^ i ^ ^ 

Similarly 

Lt ^(Z,T) » -log max p(x<,y*) 
oo i,j * 3 

and 

lit ^(XA)« -£ p(y^)logffiax p(x^/y.) 

q ( ..#00 ^ 3 ^ i ^ ® 
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3. SOME PKlPSmBS 
3.1 CootlacltT 


H XI) « iegSP^Cx.) 

^ 1 *“ ®< i **■ 


*<-l ‘<’"1 

= -log (2 P( 3 Cj^) p C3^) ) 


• - log 


( 9 ) 


viusTe 






( Sp(x ) p ( 314 ) ) 

i ^ 


(10) 


SlBoe givea ly (10) is a power meaa ©f order *^-1, 

^ 0 and we kiaow frcai Hardy , I,ittlewood md Polya (1963) that 
is a contirwoxxs fmctioa of <. Heace from ( 9 ), ]|^(X) is 
also a continuous function of °c • 

(X/X) * J log 2 P ly^/x^) 



«c-i sr^a 

a - |p(Xj^) l©gC s p(y^/x^) p (yj/x^) ) (11) 

So ]^(X/X) is a weighted mean of a finit® numher of continuous 
functions of *<1 hence it* itself, is a continuous function of «< • 
Similarly other “^-entropies associated with two Unite 
seh^es are ®ontinta>us functions of 


ly theory 16 , Hardy, Llttlewood and Polya (1962), 
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^ ClO) is a mojiotoialcally Increasing funetioii 
®f «c. Hence by (9), H^(X) is a monotonically decreasing 
f met ion ®f ®(« 

It may be noted that monotonically decreasing 
eharacter of ^^CX) can also be proved 1^ examining the 
behaviour of the derivative of ]|^(X) «<. fhls procedure 

was adopted by Varma and Kapur (referred to in Chapter l) 
for examining the behaviour of their generalisations of 
Benyi «< -catropy. 

Since by (il) H^Cf/x) is a weighted mean of a finite 
number of monotonically decreasing functions of «t, so it, 
itself, is a monotonically decreasing function of «<. 

Similarly other ®<-entropy associated with two finite 
schemes are also mono ton! caO-ly decreasing functions of 

H^(X) is maximum when all the x^ are equally likely* 
Kapur (l9d8a) proved it by the technique of dynamic progrswm.* 
ing* It can also be proved by the technique of hagrange's 
multipliers. 

Since maximiM of a weighted mean is equal to the 
weighted mean ®f the maximum values of its individual teims 
and each term in ^(Y/x) is itself an ®{-entropy for fixed x 
so by the maximal! ty of ^(X) each of these terms is maximum 
when the conditional probabilities constituting it are equal* 

Similarly the maximal! ty of other ^C-eatropies can 
be discussed. 
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3 i^ 

la H^(X) » l®g 2 p (3^) » let l®g I p (x^) 


be deaotedl f(®<) , s© 


f («) 


p (Xj^)lagp(x^) 


(12) 


I P*^(\) 

«c ^ 

2 P (X4 )(l®g p(x* )) 

Deaotiag 1 ^ ^ » tbe weighted meaa ©f 

2 p (x. ) 
i ^ 

r c< 

(log p(X|) ) with weights p (x^), by Hj, we have 


d My 2 p*^(x^)(lo£pCxj^)) (|Jp'^(x^)(logp(xj^))^)<p*^Xj^)log P(3^)! 


S P C 3 C|) 

i ^ 


u % 


M M 


( S P^lx.) f 

i * 


..(13) 


By (12) aad (13) we have 


f * («<) * M, 


yR *e Q 

f» »(*<) • — - » Mg - m; 

&K ^1 


f » ♦ » («£)« 


Mg- - 2M2^(M2 

M 3 - 3 Mg *»■ 2 
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iv 

f («C) 


n 




♦ 2H^ 


f ’ * ’ c«c) 


and so on 

f(°C) 

1- oC 



H»4X) « 


f» («) ^ f(«c) 

1 - « (1- OC)® 


^'(x) 


9 


1 

1- OC 



(«) + ^u)! 


f’W ^ f'Ct) 
— + — 

1 - 0( (l-o() 


* 5iSSL * 

1 - 0 ( 



1 

1 • «< 



S® in general hm expeoty denoting the kth derivative of 

^^CX) by 


(fc) 1 r (k) (k-Dn 

®oc * 77"; k (“<) + ^ 


(14) 


Suppose (l4) is validy so differentiating (14) again ¥.r*t. ®£ 
we have 


(k+l) 

\ ‘ 


Ot+l) (k) 
f (<) ^ f (■<) ^ 

1 * «t (l*^) 



'^nau. ^ 
1- o( 


<k-i) 
k ^ 

(iT^ 


k+i 
f (oC) 


4* 


(k) 

k Mot + 1 

1 - o( 1- « 


C f 


(k) (k*i) 

+k 



1 - o< 


1 - «< 
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(k+l) 1 


m 


1 - o( 


r(k+l) Ck)-i 

f (o() + (k+1) 


Heac® im gemeral 
(r) 


1 

l-a( 


-(p) (p-i) 

f + r 


J 


Cl5) can be written as 


(15) 


Cr) 


r-l (r) 


r-l (r-l) 


(1- o£) a (l-°c) f + (1- «<) r ^ 


(16) 


lategratlmg (i6) from t© "(g ® < *<2 ^ ®° » 

we slaall bave after a little simplification 


H 


"(o 

(r-l) r (r-l) r r-l (r) 

(1- ‘^) * ^ (1- ««3^) '►;(!-'<) f d «< 


(IV) 


la particular patting oc^ • 1 and *<2 * ^3.7) such that 

1 < o< < ©o ve have 

(r-l) r at r-l (r) 

^ (1- “t) s X (1- i^) f (x) dx (18) 

(r) 

Since f («<) Can be easily ceaputed witk tke kelp 
ef (13), so consequently can also be easily ©btalaed 

from eilker tbe recurrence relation (16) ©r tbe integral 
relations (17) and (l8)* la particular when X m bave 
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a simple relation 


(r-l) 1 r (r) 

H, « - — f (■ 
1 T I 


rCD 1 

f («t) 

^ at oC 


at ®t * 1 


UmXf Cl 5 ) 


» 1 f 

_ f ^ ^ 


, f*+ Etc 

It Etc » Lt Z 

<< -*1 «<♦ 1 1 - «< 


f"(i) + % 


« - i r'(i) 


Again putting x » 2 in Cl6)> taking the Unit as 1 
anS making us© of the value of obtained above we shall 
have 

<* 1 til 

H, » - 1 f (1) 

^ 3 


So we expect 


(k) 1 (k+i) 

f ( 1 ) 

^ k+ 1 


Cl9) 


Kow putting T * k 4 -i in ( 15 ) and taking the limit as ®c 1 


Ck+i> 

It H, « It 


oC -•i 


«a( -»1 


(k+ 1 ) (k) 

r («) ♦ 

(1 - 



(by 19) 


c 


V» 2 ) k +1 

f (1) + % 


C-1) 


SB 


1 

1»2 


k4-2 

f (1) 


Q.E.D. 


Since b®tb ]B^(X/X) aad ^^(X/Y) are weighted means 
of <^*-entropie 8 of the sort ]^(X) 30> similarly as abovei 
relations involving successive derivatives of these respective 
conditional ^(-'entropies can be developed. 

For Second De i^lYative 

I d 1 */ 

E^{X) ( log S pVi) ) 

d «c l-«< i 


1 

» — 
(l-«t) 


log S p®^(X, ) + 
i ^ 


1 

l-«c 


^P*^(3^)logp(Xj^) 

I 


(1-«C)^^^*CX) » log I + Cl-<) 




f F 1®8 Pi^i> 

1 


J P^(nj^)log pCx^> 

I p^Ca^) 

p*^(Xi)log p(Xj^) 

I ' 


p**(x^)(l©g P(xj^))® (^/Cxj^)logp(x^)) 

I p‘^(3Ci) (2 pVj^) )® 


+ (l-««) 



- 2S 


H2p^Cxj^)l®g pCx^)) 1 


.. ( 80 ) 


Bttt by CaTJOiiy’s inequality 


£ 

i 


2 


p (x^) 2 p^Cx^Xlog pCx^)) ^ (S p (Xj^)l®g p(xj^)) 


S® whea 0 < «c < 1 , by (20) we haTe 

-2(l-«) ^ ^ (1- «f h’^ 0 


&T 


^ > 


2 

1 - « 


and when «< > 0 we have froa (20) 


» 2(1- «) + (1 - -C)^ H 4: 0 (21) 

er I 

2 Ec 

Hit 4 (22) 

^ 1 - o( 

Whm all tbe p(^) are equal we have l'e(* 0. 

f I 

(20) gives a negative lower bonM of Eo( for 
0 < oc < 1 and (21) a positive upper bomd of E*^ for ®c> 1. 
Tbe fact tbat is aer® wbem all tbe p(Xj^) are equal 
shows that these bounds are precise in the sense tbat they 
are actually taken on in the special case when all the p(x^ ) 
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Siace the derivative w.r.t. *< of a weighed mean of 
a finite am her of f motions ot < the mean ©f their 
derivatives) so proceeding in the same manner) similar 
results for other °(*emtropies can he obtained* 

It may he noted that the developments of the results 
of sections 2*4 and 2.6 are more or less the manipulations of 
some of the results of Beokenbaoh (1942) and Shaiad (1948). 


3 

3 


•1 


(i) SQiea 0 < « < 1 
:i^ J ensen ' s inequali ty 


Sp(x ) log Zp'^Cy.A^ ) ^ log! pCx.) p'^iy^/x. ) (2$) 

ii 4^ i>3^ ^ ^ 


Since p(x^) ^ P (x^) f®r 0 <«( < l ,s© (23) can be written as 


2 p(x. ) log E p‘^(y4/x. log S P“^Cx 4 ) y^) (24) 

i ^ 3 ^ 1)4 ^ ^ 


Multiplying (24) by — ) a +ve nmber) wo have 

1- o( 


E!o((YA) ^ I|<(X, Y) 0 < «< < 1 (26) 


(ii) Whm «< > 1 

Since for «<J> 1) | p'^(y^/ 2 ^) ^ 1 so we shall havo 

I p(x^) i©i^ p^(yj/xj^)> f pCXi)|| i®g|jp^CyjAj^)] 


• a # 


( 26 ) 
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But X. l&g X til a ©oavex fmction ®f x | so takiag 

^ * S f (26) aad Using Jensen's InequaLityt 

(26) reduces to 



so . 

I p(x^) log 2 


Multiplying (28) by — , a negative number, and using tbo 

1 - < 

faot tbat P ^ 1 we sball feave 


H^(X/X) ^ (X,X) «;> 1 (29) 

CmabLning (26) and (29) w® sball have 


i[^(X/X) ^ ^(X,X) > 0, «( ^ X (30) 

In tbe limit, as 1, (30) tends to H^(Y/X) as 

already known to be true, so including tlais symbolically wltii 
(30) we bave 

<>0 < 31 ) 


s ^a,t) 
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Similarly ve cam prove 

S ^CX,y) «X> 0 <32> 

3*2 BGt*'iiF@e& ^irCXfY) iUscI 

Ci) wtn« 0 < « < 1 

2 > (2 * P*^<X|)C2p(yj/x3^))'^ 

« p'^CXj^) • (33) 

Smamiag (33) over i, talcing logs and then multiplyiag by 
.,, ^ . ^ a +ve no,, we shall have 

1- o( * 

I^<(X,T) \{X) 0 < *< < 1 (34) 

(ii) When 1 

5 ^ ^ P(Xj^iyj))"'* p“‘( 34^)(2 pCy^/atj^))"** 

5 4 

summing (35) over i, taking logs and then multiplying by 
negative mm her, ve have 

i(^(X,y) > %(X) «£ > 1 (36) 

Combining (35) and (36) we have 

«<(X,I) > E^CX) «< > 0, «< 4 1 (37) 

In the limit , as (^) tends t© %(X,y) > %(X) as known 

to be true, so oomMning tMs symbolieally with (37) we have 

« (X.Y) >y HL,fV\ 



• 29 “ 


Similarly m cam proT® 

3*3 Bctvoea lic(Y/£) AM ^<(y> 
(i) When 0 < °c < 1 


By J'ensem's inequality we have 
2 P(x^)l®g I p^(y^/2:^) ^ log I p(ac^) j p”^(y^A^) 

-logs 1^2 P (^) p‘<(x^,y^)J 
la 2 p(ac^) p“^(x^,yp let l/k * «< ^ad lA* « i- « 


(38) 


«o that (lA) + ClA») « 1 and spply H5lder‘s inequality 

for k ^ 1 , th» 18 j page I4j jferdy yLittlewood and Polya (1952)^ 
w« have 


•'* oc l-*< 

Zpix^) p <a^,y|>^ (j p(3^)) (2p(X|^»y^))®‘* (39) 

Suhstitutiag (39) in (38) and dividing the result by (Vl*^), 
a positive number, we shall have 

%((X/Z) ^ BLcCT) 0< *< < 1 (40) 

(ii) When 1 

Sl*c« 2 t'ir/xi) 4 \ , aa 
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Since X log X is a convex function ©f x, so talcing 

H “ I p applying Jensen's inequality, (^// ) 

reduces to 


S pCx^ )log2p‘<(yj/3c^)^ jipCx^)^?*^ > log jspCxj )|p"'(y^/x^^ 


^ <yj/x^)lo^ (x^)p‘^(Xj. ,y^) 

« # # e ^ 42 ) ^ 

j^ince p(xj^) > P*^(x^) for ®t > :^ 

But 2 p (Xj^ )p ) *s 2 p*^ (Xji^ »y 4 ) ^^ 1 , so letting 

Ifi * i|3 ’* 

®( 

2 p CXj^,yj.) * p ^ 1 , (42) reduces to 
i,i ^ j 

<a( p ^ -«--i 

2 p(x^)log 2 p Cyj/3^)> p log I 1^2 p CXj^) p'^CXj^jy^^ (43) 

®C o{ 11 

In 2 p (x^) p (x^,y^), let £ » «< and i-«( , 

so tBat ’^4' a 1 and applying H51der‘s inequality for k < i, 
th* 13, page 24, Hardy, Littlewood and Polya (1952) we have 

1*^ at 1-oC oC -f 

2 p (x^) p (x^,yp> ( 2 p(x^) ) ( 2 P(x^,yj)) « p\tP 

Hsing this in (43) and multiplying the result so obtained by 
(l/l-o() , a negalsivo number, we shall have 




( 44 ^ 
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Simce p 4 1 » 8® frm (44) w® kave 

H^<YA) 4 *< > 1 (45) 

ComlslBiiig (40) aM (45) w® kav® 

H^(Y/X) H^(I) «> 0, «( ^ 1 (46) 

In tke limit, as *< *• 1, (46) tends to H^(T/X) ^ H^(T) 

as already know as Skannon^s inequality, s® ocoakinlng 
tkis symkolically witk (89) we kave 

E^(yA) ^ H^(Y) «(> 0 (47) 

Similarly we can prove 

(Y/Y) ^ ^(X) *<>0 (48) 

3.4 A General isation Of Khinckln*s generalisation Of 
Skanaon»s Inequality H,(Y/X) ^ H^d) 

Ktdnckin (1967) gave a generalisation of Skannon's 

result 

-i2^P(Xiiyj) l©fi ~ f 

tkat states tkat (49) is true if in it we sum over only a 
certain values of the subscript but of course over tke same 
values of 4 on botk -tke sides, m shall prove tke same 
result for (gtu^ inequality given lof (46) i.e* we shall prove 
that tke inequality in (46) is valid whether tw ^ 
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in it form & complete scliaae or not. Im brief It mLII be 
proved that 


S p(x4 ) log S p (y/aE^) ^ log 2 p (y.) (SO) 

l-o( i <* 1 - o( Jal •' 

*< > 0 , «< f 1 

* 

where Z demotes swmat^om over certalm values of J. 
j*l 

(1) Whoa 0 < « < 1 


Sy JTeasea's iaequalltsr we have 


2 pCx.) log 2 p‘^(y./x. ) ^ log 2 p<x,) 2 p^ij./x.y 

i ^ 3«1 J ^ i * 3*1 ^ ^ 


l-« 


® log 2 2 ^ p ix^) p°^Cxj^,yj) 


log 2 p'^Cyp 2 p'^(xj^/yj)p^(Xj^) 

1 jL ___ 


( 61 ) 


l-o£ 


la 2 p (x^) p‘^(x^/yj);,let - a « 


* 1 “ *< so that 


Jl 1 

^ + — « X ahd apply Holder' s inequality for k > 1, we shall 
k k' 


I P (Xx> p (x^/yj) 4(2 f(Xj^)) ( 2 pCx^/y^)) a 1 


have 
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Let p ^ ^ 1 and p » nax Pj 

Substituting this in (6i) m shall have 


at ^ 

2 pCx. ) log 2 p (yi/ac, ) 4 log 2 p ( 7 .) + log p (52) 

i ^ ^ ® 

Multiplying (62) by (i/l-«<) , a +ve number, and using the 
fact that p 4 1 shall have CbO) for 0 < *< < 1 ♦ 


When > 1 

Since 2 P^Cyyx^) ^ 1 , so 
4 i ^ 


* «< 


Z p(:^)l®g^2^p (y/x^)^ sp<x^) r ^p-CyjAj) 


4»1 


log 2 )(63) 

^*1 J * 


Since X log X Is a convex function of x, so talcing 
* «< 

Xj^ * 2 p (yj/x. ) and applying Jensen's inequality, C&3) 
reduces to 




log 


2p(X-)2 p*^ 
i 1 5»1 

L ■ ‘ 



CM) 



this and a little manipulation would reduce (M) to 



(56) 
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l ****®^ ^ 11 

la Zp (Xj^) P (x^/y^) f let al-«c and 

apply Solder’s inequality for k < i , we shall have 


1 -«C 


l-«sC 


p <3E^) p Cxj^/yj) > ( z p(xj^)) ( z p(x^/y^)) = l 


l*>oC 


Let Z p (x^) p (3Cj^/yj) “ > 1 and '2^* min so 

i 

substituting this in (5b) we shall have 


ZpCxj^) log z ^p (yj/xj^)> P log 


"1 


p (y^)+log2J 

a ^ 


(56) 


Multiplying (56) by (l/l-oc) ) a -ve nmbery and using the 
fact that P ^ 1 and '2^^ 1 we shall have (50) for «< > 1. 

In the limit , cas < 1> (60) tends to Khinchin’s 

ZiSCL 

generation. 

Similarly proceeding as above we can prove 

] p p [ ^ 0([ w )|t 

i Z p(yn) log z^ p (xi/yjx i— log Z p (x*) (57) 

l-e( i ^ J l-«t i*l ^ 

^ ^ o , I 

In the limity as ly (57) tends to EMnchin's gmerallsa* 
tion* of S H^(X) 

3.6 Study of Relations Between ^(X,Y) iai ^(X) + ^(Y/X) 


Two infinite sets of inequali1d.esf one for the range 
0 < < < 1 and one for ly each tending in the limit as 
«< -• 1 to Shannon* s result %(X,Y) » H 3 ^(X) ♦ Hj^(Y/X) have 
been developed. Two bounds for ]^(XyY)y one t^per for 
0< «< < 1 and are lower for ®t > It have also been developed. 
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Cas« (1) Wli«a 0 <«C < 1 


1 

F 


“C- ^ l/k < 


log 2 p * l®g S P (xi ) P (X|^) P 


1,4 


Wiser® k is so cbosea that 0 < k < i 

, «c. . 17 r c< 

log 2 ^p (xjL»yj) * log s j^p (Xj^) J|2 p (x^)p (yj/xj^) 


JU 


j 


( 68 ) 


1 1 

Claoosinig k* stibject to “* + * =1 aod takiag 

k k' 

l/k oc «(-l/k 

2 p (x^) p m4. p (x^^) a In 2 a^ ^ 

lA k» l/k' 

(2a^) (2bj^ ) ^ the result of H&lder inequality 

i i 

for 0 < k < 1 , (SS) reduces to 




log 2 P (x^,yj)^ log 

1,4 


I r “(“l/k ■) k' 

log 


l/k* 


f l/k 'll 

E |>| P Cx^lp (y/x^)j 


lA 


1 (‘<-Vk)k* , 

-,log Jp (x^) + * log |p(x^) 


r o( 

sp (7j/x^) 


(69) 


But by yeiise]i*s inequality 

r 

log fp(x^) I p*^(y^/3^) k j pCx^) log |p^(y^/x^) (60) 
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Using (60), (69) reduces to 

«C 1 c( 

log I (Xi,yj):^-, log S p (x^ ) 4- ? p(Xj^)log |p (y^/x^)(6i> 

Multiplying (6i) by (l/l-«<), a positive number, 
and malting use of the fact that Cl-°<)k* * (j/k + l/k* - oC)lt* 

» 1 * (o( - i/k)k*, we shall have 

H ,(X, t) H (X) + H.(Y/X) (62) 

(o(*l/k)k« 

0<oC<i, 0<k<laBdl/k4- l/k» » i 

Under the conditions for the validity of (62), 

(« - i/k)k' lies between 1 and o® since 

1 ee 0(l£ 

( o( - lA)kt » (63) 

1 - k 

For 0<«C<1, 0<k<lwe have 

«< k < k 

or «<k-i<k-i 

or 1- • > 1 . so from (63) ('<-i/k)k' > i. 

So from (62) accrues an infinite number of inequa- 
lities connecting I(^(X,Y) and ^^(Y/X) for 0< < < 1 to 

H^(X) for As 1, all these ineqiialities tend to 

the Shannon’s equality H^(X,Y) » H^(X) + H (Y/X). 
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Case (11) When 0 < «< < i 

Bewrifeing CS8) on the assimptlon that 1, 

we have 


log 


Z 

US 


• \ 



( 64 ) 


Choosing k* such that l/k + lAc* = 1 and applying EBlder 
inequality for k j> 1, (64) redluces to 


log 2 p log 

1 1 ^ 




Vk« 

4* log 


j r 1/k ^o( 

2i2 p (X )p (y./x,)( 
, . . ( 66 ) 



Hultiplylag (66) by (l/l-®<) , a +ve number, we have 


H^(X,t)4 H (X) 4- 

^ («<-l/k)k« (l*«()k 


log 2 p(Xj^) 


s p (y. 


/*i)l 


k 


(66) 


0<«t<l,k> landl/k + i/k» =* 1 


Taking limits of (66) as k *• ®Owe have 


It 

00 


fi (X) 

(«-l/k)k* 


* 1^(X) 


and Lt 
k *♦ 


1 

(l-«<)k 


log 2 p(x. ) 
i ^ 



k 


and 


( 67 ) 
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oo 


/7T^ f 5 Cy./x») 

L 


( 67 ) 


ipOCi) 

j A. 


= Lt 2 
ooi 


, k ; ^ 

2 p 1^ ^s| i^Cyj/3^) 


(68) 


2 p(z. ) ' 2 p*'(y^/ic, ) 

L 


_j 


SHMQe the expressloa on the a.H*3. of (68), whose 
limit, as k ^ , is to be takoa, is a weighted mean of the 
quantities (l/l**<) log 2 p'^Cy^/x^), so it should be less than 
max j(l/l-«c) log I p'^Cy^/xj^^ . Hence using this and (67), 

(66), in the limit as k -• , reduces to 


I^{X*I) ^ ^(X) + mix [(l/l-«c) log 2 

i L j ^ 


or equivalently 


H^(X,Y) ^ ^(X) + max. ^(I/x^) (69) 

0 < *< < 1 


Note that (69) gives an T^per bomd for ^j{(X,Y) 
for 0 < “=< < 1 . 


Case (i) When°c>l, 

Note that in deriving (6i) no condition on « was made, 
so multiplj^ng it by (i/i-®<), a negative number, we have 


%(X,Y) ^ I ^ (X) 4 %CY/X) (70) 

(oc«x/k)k* 

«C^1, 0<k<land l/k 4 l/k» « 1 



We observe that («( - l/k) It* <1. Under the given 

conditions for which. (70) is valid* and 

s*! 

k . How thwe are two cases (1) k > l* (ii) k < 1* 
In case (i) ('<- lA;)k* is negative and in case (ii) it lies 
between 0 and 1* 

So frtM (70) accrijss an infinite amber of inequali- 
ties connecting Ha<(X,Y) and ^^(X/X) for *<^11 ^ ^ 

E^(X) for 0<«<<1. As«<’*lf all tt^se inequalities tend 
to the Shannon* s result H^(X,t) « \(X) + H^(Y/X). 



. Since in deceiving C6S)|no condition cai < was made, so 
multiplying (66) by (l/l-=<) * a negative amber* we have 




(X)4- — 

(«=<-l/k)k» (i-*<^k 


leg Z p(x^) 


£ p 

i 




ik 


(71) 


Taking limits of (7i) as k-* ®®we have 


Lt H (X) « H.(X) 

jc -» oo (cc«-i/k)k* ^ 


(72) 


and 

Lt ^ .. log 2 pCk. ) 

<SO (l*c<)|c 1 3. 





3 ® 


a ^(Y/x^) 


( 7 ?) 



Using (72) and (73 )| (7i) reduces to 


^ H (X) + min. H (T/x ) oc;> i (74) 

(74) gives us a lower bound for 1^(X,Y) for ®C/> 1. 

Similarly > proceeding as above, we sl^ll have for 0 < «< < 1 

1L(X,X)^ H (T) + ^(X/X) 

0< Ti:< 1, lA ♦ iA» 1 

H^(X,X) 4 \iJ) + max. (2/y^) 

and for ‘<J> 1 

^(X,Y) < H (Y) '►^(X/X) 

^ (a<-i/k)lE* 

0< k< 1, lA + 1/k* 3 1 

H^(X,X) ^ ^(T) + min. ^(X/y ) 

J * 

3.6 A Study of Xbe Validity Of **^(X,Y) « ^(X) 4 ^(Y/X)'* 

It will be proved tbat when the ohannel is either 
symmetric or binary erasure, the result ik(X,Y) « (X^+Et((Y/X) 

for < ^ 0 will hold but it is not true in general, 

tlift ,l,g„ sMbljis, 

Since the channel is symmetric if each of its rows and 

each of its eolimns consists of smse set of elemmts, may not be 

in the same ordery so for such a ehannel Z p'^(y«/x 4 )» a constant 

5 3 ^ 
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i • l)2f ••••) n.* 


Bence 




* "Jir ^ p^Cx^) I p'^Cy/Xj^) 

* -i— leg s p^(x^) + log 2 p'^Cy^/x^) 

1*^ i j 

_ _L_ s: ^ -'— X ^ ^ j 

"" I - o( ^ ^ l-^(l j 

« ]^CX) + H^(X/X) (75) 

M)w.,-$fe§, ffi^iam,tt3- „ 1 a Ham siasMEi, 

Again for such a channel ^ p'°^(y|/x:^) « a constant 
for i * 1,2 .,»., n and the truth of q<(X,y) * ^(X)+^(Y/X) 
Can similarly he checked. 

Hote that validity of ^(X,X) « ^CX)4*H^Cy/X) dees 
not Imply the validity of ^<CX,Y) * ^(Y) 4- ^(X/Y) 

In general 

^<CX,Y) 4 ^<X) +^(Y/X) 4 VYK ^(X/Y) «> 1, « 4l (76) 

5i;$pose that the symbols 0 end 1 are tran^itted 
with respective pro hahili ties 1/3 and 2/3 and received as 
either ae- 0 or 1 or 2 ^th the follo%4.ng noise characteristic 
of the channel; 

J 3 g 


NJL 
X 

0 

1 


1/4 1/2 1/4 

0 2/6 3/6 
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Calculations would give us 

pCY = 0) = 1/2, p(Y*l) a 13/30 and pCY=2) = 29/60 

1/12 1/6 1/12 

p = 

0 4/15 2/5 

and calculations majie by taking logs to the base 10 would 
give 


%(X,Y) 

s .5642 

H^gCXjY) a .6382 

%(X) 

» * 2552 

Hi/2(X) * .2884 

%(X) 

S .3682 

e 

11 


« .3313 

BL^2(Y/X) * .3525 

H2(X/Y) 

= .2521 

Ej/2il/Y) a ,2460 


(76) caa be easily verified for » 2 and ®c » 1/2* 



Consider r finite discrete sobemes specified 
respectively by the discrete random variables 
Let the product scheme obtained by Joining these r schemes 
be denoted by ••.•<£) 2^ . In tMs product scheme 
* • ® Xy, following are a few generalisations of the results 
developed in section 3 t 

(i) the inequality in (47) will get generalised to 
inequalities of the form 

^a<^Xj/X^,Xg, • • , Xy»^) ^ ^(X^/^f • • (77) 

(li) fbr 0< « <1, the inequalities in (62) will 
get genersCUsed to inequalities of the form 
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H (X-) + H (Ssf/X,) 4- 

1 y («c-lAi)kJ^l (oc-iA2)k|® ^ 


Ar H (Xy-i / Xi, X*.2 ^ 

+ /X^yXg) • • • •Xjia.^y 


(78) 


wkere 0<lc^»,.,, kr.2^< 1 aad l/k^ 4- l/kj^ * 1,..., 

l/fc’y.j. * ^ Ineqtiallty la (69) to inequalities of 

tke fozm^ 


*1 

+ ma3c* l^(Xg/x^,X2> 


4- max. H<(X^ X ,.*.,x ) (79) 

^l»**»*r*l 


(ill) Ibr I ,the laequaXitlos in (70) will get 
generalised to inequalities of the form 




I O « • 




♦ H 


(«(-lA2)k^ 


(X/X^)4-, 


4" ^( • • • •> y Xp**l) (80) 

where 0 ^k^}««**f ^^*■2. ^ ^ end l/k^^ 4* l/kj^ l}**«»f 

4- ^ inequality in (74) to inequalil^es 


of the feim 
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+ mia. ^(3^/x ) 


H* ml®. Il^CXg/x^^jacg) ♦. 


^1>^2 


■+ ml®. !I^(X C81.) 

Xj^l . * • t^y«3^ 

e i . 9ffi i....oi:, C77) t 

On the assTMption that an event (Xg, . . . of the 

prodnct space 5l^® .... ®Xjwi occnrredj lot the prohahl- 
11 ties of the events of the type (x^, belonging to 
he given 

p(x^,Xy /xg,..,.^ Xp^3^ ) » 

and the schmes given hy the random variables X^ and X^ become 
the schemes givma by the randcaa variables X^ md X^ with 
pro hablll ties of the type 

pCx^/xg, ...., = q(x^) 

pCXy/Xg,..., x^^) * q(Xy > 

l&f (46) for ot ^ 0, ®C 4^ 1 » w« 

^(x; / Xj^)s V^’ ^ 

Using (83) and (84), the h.H.S. of (86) is given by 

q(x3^) log 



BoiiX* /X, ’ ) » -i- 2^ 
^ •*• 1 - «< Xl 


^83) 

(84) 


( 86 ) 
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XTsi^xig C83) C^d) v® 


qCXyAi) 


p(Xy>X^/Xg,.>.,Xy.^) 


9(Xl) 


pCXj/Xgj»*«j ^p—2, ^ 


* • • • »3 Cj,^ 2^) 

Using (87) and (83) , (86) reduces to 


(87) 


1 o( 

^ P(Xj/X2,...f*j,.3^)l©g X p (x^/X2^,22»**l3Cy.3^) 

... (88) 


Using (84) 


H (4) = -— 

•* 1- =C 


I !> U/X3,..,i^j^) 

JSk 

T 


(89) 


Using (88) and (89), (86) reduces to 

oc. 


•ir X ' * M3t,„-l)l®I X f (x^x^^jXg, . •-»«-- 

1 *2^ i Xj, 


IP^ 

< ^ l®g X p (Xy/Xg,... ,Xy.3L) (90) 

r 

Multiply (90) ty p(n2» •• .fXj..^) and stwming over 1^. ..®Xjp.jL 
ue stall taxo 

^^(Xy/Kj^fXg. . . . >Xjp.2.^ 


( 91 ) 

*< > 9, a( ^ 3^ 

Xn the limit, as '* 1 , (91) tends to tte tnovn result 

^ ♦, j comlaining tMs 
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mith (91) we shall have (77). 

Q. E.O. 

Qutliae of the Proof for (78) : 

Applying the procedure followed to get (62), we 
shall have 


^(X^, • . . ,Xj,) 


H 


(=<»l/l£^)k 


(X^)+ ^^(Xgi • • » ,2^/Xj|^) (92) 


where 0< k^< i and 1/k, + l/k£ = l. But in (92) 


H[^(X 2 ,..,X/X 3 l>=( 1 / 1 -«C)S p(x^)log S p'^(x 2 ,..,x /x^^) 

X^ Xg®,,,,®3(^ 

ICr i ting 

a( 

log 2 p (Xo,...., X /x,) 

X.2<& ®Xy r X 


^ =<-3^2 . . l/lto of 

= log Z p(V*l>] 2 P 4V*l) P 

Jt2' ^ A®. . . . .OXy 

where 0< kg < i , choosing kg* so that l/kg + lAg* « 1, 
applying Hblder inequality and following the procedure 
adopted for getting (62) we shall have 


H,(32,..,Xy/x )> H (Xo/x^) 

^ ^1 (oc-oAg)!!^ ^ ^ 

•l'(l/l-^)2p(x2/x2^)log2 p‘^( X3,..,Xy/x^,Xg) (93) 

3^ 

Multiplying (93) by p(x^) and summing over we shall have 
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Cofflbiniag (94) vrfLth (92) aad applying to 
H^(3^i ***fljp/X^*X 2 ) the procedure f©r getting (94) f®r 
another r-3 times, we shall have (78). 


gutllne of the Proof for(79)> 
ltc(X,,..,JL) * log 2 pV*,...,x ) 


1 

l-« 


leg Z 


“t-lAn lAi *< 

F (Xi) Pi^J P (Xg,**,Xj/x3^)(95) 

®Xjj, 


where k ^ 1 . Following the ssm® procedure as used for 

Hi* 

getting (e9)> fron (9S) v» sball kave 


Bat In (96) 


(96) 




%/x^) 

1 


— In g S P*‘(x , . . ,xyx ) 

l-o< 2 ^ 1 


o(-l/k, iAl-1 oc 
log 2 f (*2 /x-) p (xj/xi)p (Xg,.., 

!•«( *6'‘Xy 


V*i>V 

.. (97) 


Again applying to (97) the procedure adopted to get (69) 
fron (64) we shall have 



* max,^(X2/!Cj^)4'aax. ^(3^, .tfX^/x^tXg) (^8) 

Comhliilng (96) with (98) and applying tojaaX Bt((X^i ••,Xj/x^iXg) 
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ha VS (79). 

(8j&) and (^1) can similarly b« pK>ved ty repsating 
r-l times the procedure adopted for getting (70) and (74). 


&• ALTERMnYE DEE[MI;Q1,„0 F GQMnLTiQN iL. °C-5mHDPI MD 

immAwm- 

There is another vay of defining conditional “f-entropy. 
Instead of talcing the weighted mean of :i^(Y/xt,), the way we 
did for deriving (S) from (4), we can also consider again 
the EoImogoroV’-Hagnmo generalised mean of the *infomation 
contained* variable ^(X/a^) w.r.t. g(x)r «c>0,oc 4 1 

as we implicitly did for getting ^^(T/x^) or Benyi (1961) did 
for getting ]|^(X)* So if we do this way* the conditional 
^•entropy wonld be 


4((X/X) * g 


X p<2;4 ) 8 

i * 


(1-x) W^(X/x^)j 


Bat by (6) 


(99) 


H^(X/x^) * log X p‘^(y^/3i^) , 

so 8Ul»titating tMs in (99) and simplifying we have 


]^(X/X) 


log 


X p(x^> p CT|/x^> 


i * 1-3 

«(> 0 ^ < 4 1 

Similarly following these lines «t-eq,ai vocation wild be 


(100) 


MIX/S) * — ^ log 
^ 1 - sc 




( 101 ) 



In the limit, as 1, both (lOO) and (lOl) tend 
to the corresponding Shannon’s definitions. But toy J’ensen’s 


inequality 

log ^ I P(3Ci)log E p*^(yj^/xj^), 

so for 0 < oc < 1 


a( 

JL. 2 p(x, )log I p (y/x ) 4: 
i-wc i ^ i 



and for ^ 1 


( 102 ) 



• • * C103) 

By (102) and (103) conditional “(-aatropy given toy (6) 
i s less for 0< «< < 1 and more for ^ 1, than that givaa toy 
(lOO). Similar interpretation' can be had for (6) toy ccanparing 
it with (101) toy Jensen's inequali^. 

Results idiich involve specifically the definitions of 
Hb((X/X) or 1^(3?/X) and given toy the following equations are 
also true if we use the alternative definitions of and 

given ly (100) and (lOl) respectively s 
dll the results, except those given toy (62) and (70) 
and their generalisations ^ven respectively toy (78) and (80), 
are algo true for these al-ternatlve definitions of conditional 
«t-entropy and «c-equivocatlon. these can be proved for the 
definitions given toy (100) and (lOl) toy simple manipulations 
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of the procedwes adopted to prove them for definitlohs 
givm by (5) md (&)» Itor instance the resialt in (47) 
for ^CY/X)^ given (l00);can he proved as follows * 

Itor 0< *>( < 1 , comparing (38) and the definitions 
of ^(Y/X) given by (&) and (lOO) and retracing the steps 
for obtaining (40) from (38), the restalt follows* 

For «< > 1, ty Jensen* s inequality we have 


log £ p(x^) £ ^ 2 P(x^) log S p'^<yj/x^) 


and further retracing the steps used for getting (46) from 
(41) we shall have the result* Or ^se 



By the argument used for getting (44) frcma (43) we have 


at 

s p i\} ^ P Cy^) 


(106) 


Using ('<’5'), (l-^.H) reduces to 


log s pCx. ) p'^<y/3t ) log f p'^Cy.) ( 106 ) 

i,J * d a. i J 

1 

Multiplying (106) by ------ , a negative number, we shall 

1 - 

have the result* 
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1. Mlffi M m OI. 

Since of-entropies defined in section 1 of chapter 2 
are associated vith tvo finite discrete schemes* so tiiese 
are also the =<-entropies associated viith a two-dimensional 
discrete memoryless channel with input and output specified 
respectively hy X and X or a commianication system with two 
parts. Transla'I^ng these for such a channel we have 

(i) ^(X) Is defined for the input X. 

(ii) i^(Y) Is defined for the output T. 

(ili) I|^(X,T) is defined for the communication system 
as a whole. 

(Iv) ^(I/X) is defined about the output where it is 
known that X Is transmitted and may he termed as conditional 
«c- mat ropy. 

(v) ^ 5 ((X/TC) is defined about the iiput where It is 

knom that Y is received and may be termed as "^-equi vocation. 

The results givma by (47), (48) and (76) of chi© ter 2 
make us define two types of «<- informations processed in the 
channel and hence two types of <<-capacities, edundancl es 

etc. and these concepts are defined in section 2. In section 3, 
“(-information, wbsn, corresponding to a single input, a number 
of outputs are received, is defined and the available result 
on the additivity of mutual information is generalised. In 
section 4, following McGill (1954) “(-informatim of several 
alphabets is defined and expressed in terms of entropies. 


•■1.1, T, S<*“ 


■ :.v-j 


JS82L 
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information through cascaded channels is generalised. In 
section 6, behaviour and capacities of five varieties 
of channels are considered. In section 7, the validity of 
the results of this chapter is exaaiined in the light of 
second definition of conditional =<- entropy developed in 
section 5 of chapter 2. 

2. °C.li:H)rMAflC?IS.°C~CAPAGITIES. «(~BEDUSrDieTCIES MD °C«-EFF£CIEITCIES 

2,1 l~Xafpipatl9.n8, 

We may define information, 0, 1, conveyed 

about X by Y as 

I^(X; Y) = H^(X) - (1) 

and '^-information, ^ 0, < ^ 1, conveyed about Y by X as 


I^(Y }X) * ^(Y) - ^(Y/X) C2) 


Note that, by (76) of chapter 2, in general I^(X;Y) ^ 
I^CY-jX) and, by (47) and (48) of chapter 2 , both I (Y;X) 
and Igj(Xj Y ) are positive. 

¥e may term I< 5 ((X 5 y) and as '^-informations of 

type I and type II. 


In the limit, as - 1, Io{(X}Y) and I^^CYjX) tend 
respectively to I^(XjY), information conveyed about X by 
Y, and IiClj X), information conveyed about Y X , Shannon 
(1948). But l 3 ^(X;Y) « I^(Y 5 X), so, as 1, both I^(X;Y) 
and I^(Y; X) tend to I^(X;Y) , Shannon's definition of 
tran^ormalion. 
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Sapasi,ttg,s 

fwo i^pes of o<-*infoimations> defined alsove, in a 
natural way give rise to two types of “C-capacities of 
type I and type II. capacity of type I may be defined 
as 


( 1 ) 

a max I^(X}Y) a max 

P<x^) 



(3) 


and capacity of type II may be defined as 


( 2 ) 

a max 


1^(15 X) * max 


p(x^) 


H^a) • 


Ii(<l/X^ 


(4) 


Both (3) and (4)> as indicated, are to be maximised 
over the input pTObabilities* In the limit, as *< -* 1, both 
tend to the Shannon’s definition of capacity. They may be 
interpreted as the maximw of ^^-informations processed by 
the channel as perceived fr<m the sending end and the 
receiving end re^ectively. 


2*3 «<-ledundancies And «< -Sfficlenoies. 

Two types of «C-lnfoimatlons and «C-capacitles would give 
two types of *<Tedundancies, *< >o., ^ 1 , both absolute and 

relative, and two types of «<-efficleacies «( > 0, «t ^ 1. 

U) Absolute °C*Redundancies 

Absolute ‘^-redundancy of type I may be defined as 


(Abs.Bedundancy)^ » Max I^(X|T) - 
and absolute ‘^-redundancy of type II as 
(Abs. Bedundancy) 


< 6 ) 


■ Mj»X 1 .(ViT^ - T fT«Yl 


/ei\ 
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In the lialtf as li toth (S) an<i 

' ten 3 

known definition of absolute redundancy, 

Cii) 

Belative ‘=<-redund®acy of type I may 

uefined 


to the 


B.S 


(Bel»leduna,aney)'‘' « 


(4bs. ledundancy 
Max I^(X; Y) 


( 7 ) 


and relative “(-redundancy of ^pe II as 


2 

( Sel, Bedundan cy ) * 

«< 


( Abs. Bedundanoy) 
Max IocCYjX) 


2 


C8) 


In the limit* as 1» both (7) and (ft\ 

^ tend 

kno^ definition of relative redundancy, 

(iii) "(-Efficiencies 

“(- efficiency of ^e I may be define 


to the 


(Sfficieacy) 


Y) 

Max I^(X5Y) 


C9) 


and "(-efficiency of type II as 


(Effldcney) 


2 


(YtX) 

MsX I^(YjX) 


In the limit* as "( ■♦ 1* both <9) and (^qj 
knovn definition of efficiency, 

(iv ) Bel^ation, Between ."(— Effl den 


( 10 ) 


tend to the 




ly (5) »(7) and (9) we have 



« oo - 


and (©)» (8> and (lO) -m iiav® 


8 2 

(a«l»Bediindancy) * 1 - (Efficiency!. 


^MmMAUlQM MD ASiaTIVE PBQPEBTy 

fhe sitnatlon wben an input may te received ty a niMlser 
of outputs ean arise (i) when tbe input symbols of a noisy 
channel are rep®ate<2 a niMber of times rather than transmitted 
just once (ii) vdaen the response to a single input is a 
sequence of output symbols rather th^ a single output symbol. 

Sxjppose for each irq>ut symbol from the input alphabet 
X » l*l»...in we receive two symbols y^^j and yg^t 

yj^j from the output alphabet ^ ® eud 

ygk output alphabet Tg » \y2k.] !»•••» mthout 

any loss of generality we may si^pos® that the two output 
symbols are received in the order y^ t 72)s.* %0J3i bhe 

rec^^tion of the first output symbol the priori* probabilities 
p(xj^) of the input symbols change into the <a posteriori* 
probabilitLes ah^ t^on the reception of the second 

output symbol pCx^) change into the 'even a more posteriori* 
pro baMli ties pixj^ / y^^j, So when two symbols y^^ and 

yglj. are received the *<-«atropy of the input symbols change from 


^(X) u -i — log 2 (x. ) 

^ 1 - «*( i * 

to the *a posteriori* ^ entTOpy 


«< > 0 , «< 4 1 


log E p^x^/yy ) 


( 11 ) 


qC '> 


^ JL ^ 
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and then to the »even more a posteriori’ «t-entropy 

^^CVyy. ^ i°g f »°‘(Vyii'yak’ ■<> °>‘< ♦ ^ 

... ( 12 ) 

So averaging <ll) over the we shall have «c-equi vocation of 

« I p<yy) ^<2/ y^j) (13) 

and similarly on averging (12) oyer all the pairs (y^, ygj^) 
we shall have equivocation of X w.r.t. and Y 2 

H^(X/XpY2^ * 

(l3) and (l4) suggest two ways of measuring '^-information 
of “type I i.e, information ctaivey^ about X by and Yg t 

(i) ¥e might define “t-information conveyed about X 
by (Y^«Y 2 ) Just as we did when the channel outgput consisted 
of a sin^e symbol l.e» 

«<> 0, «C 4 1 (16) 

(ii) We might consider the maount of ®t-infoimation 

provided about X by Yj^ alone and then the amount of «c-infoima* 

tion conveyed about X aftec we have seen Yj^i this leads to 

considering 

H^(X) - {m^} (16) 

and 

H^CZ/lTj^) - ( 17 ) 

(IS) hag already beea defined as I^(Si Tj) fiy (1)1 . 


It is 



I^CXlTg/X^) » X2^ «C> 0, «C ^ 1 CIS) 


and ¥» may call this as o(*information conveyed about X by 
Yg after we have observed Y^^* 

]E^ (77) of dbapter 8, both (16) and (IS) are positive 

and in the limitj as 1, both tend to the corresponding 
concepts of mutual information dealt \idth in ibramson (1963). 
Adding (16) and <18) and using (iS) we have 

I^CXjY^tYg) » I^(X5Y3 ^>i- I^fCXjYgAj^) *<> 0, ^ 1 (19) 

(19) eatresses the additive property of *<»information as a 
generaili sation of additive property of mutual information 
since in the limit* as 1, it tends to the corresponding 
known result given on pa^ 126| Abr^son (1963). So ctmiblning 
tMs known result azs3 our generalisation we can formally state 
it as 

Yh. 1 Ihe average '<*info 2 Sttatlon ^ 0 of type I conveyed 
about an iz^ut by a compound observation does not depend on 
whether we consider the compound observation as a whole or 
broken into its ecmponmts* 

The proof of the theory when a compound observation 
consists of more than two elementary observations follows 
from (19) by induction. Suppose that the theoraa is true 
for m < n elementary observations constituting a ecaapouzsi 
observation, so when n » m ♦ 1, by (19) we have 

• * f ^i'^4l) ^ lo((X5Xj^, • • • f)^) 

( 20 ) 


But ly supposition 
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and this reduces C20) to the res\ilt peq.iiired. 

The addittve property of “t-inforoiation of type I given 
In section 2 makes us consider a sequence of “C-entropies, 

=<> 0 , « 4 1 


Each memher of this sequence is no greater than the proceeding 
one l^by (77) of chapter ^ and the difference of any two 
successive members can be regarded as the ^infojsaation conveyed 
about X provided by a new observation s 

Ic<(X;Y^) * ^^U) - 

I^CXjYg/Yj^)* H^(X/Y^) - 

I^CXjY^) is the ^(-in formation conveyed about X by Y^^I^CXjYg/Y^) 
is the ®t-infoimatd.on conveyed about X ty Yg after we have 
observed etc, 

Allowing HcGill (1954) we can define '=<- inf ormatton 
o( )> 0, ^ 1, of more than two alphabets* Fop three alphabets 

X, Y^, Yg we may define 


I^(X;Y^;Y2) * * ^c<<X5Y/Y2) * H^(X)-i|((VY3^)-l<(X/Y2)+ 

Yg) (21) 
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In tlie limit, &s ^ ■♦1, (2i) tends to a kno-Ma. result given 
in AtrsPson (1963). 

It may be noted that for < ^ 0, <=< f 1 , 
unlike I^CXjYj^i ^ 2 )% not sysm-etrical in its argximents 
hut, like l^(X|X^j Xg^ cannot he positive in general. 

The above notion can !:» similarly extended to more than 
three alphabets* 

5 . smmjLsmmi 


¥e assme that a charnel with its iii>ut symbols X 
i » l,*..,n and output symbols ^ 5 * l,....m is 

cascaded with a second channel as in the figure below 


;x 


— >■ 




Fig* 1 

with its input symbols identified with Y and outqput symbols 
consisting of Z » ^ “ 1, •*• ,t • 

These cascades give certain relationships among symbol 
probabilities* When is transmitted, let the output of the 
first channel be y^ and let y^ produce from the second 
channel* The symbol depends on the original symbol 
only through y^ so the knowledge of Intermediate symbol y^ 
makes the probability of obtaining the depend only on yj 
and not on the initial symbol which produces y^* This 
relationship may be written as 

p<%/yj,Xj^) a p(s^ /y^) for all i, U ^ ^22) 


Since the probabilities pCXj^/y^) are known as ‘a priori* 
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probabilities aad (22) makes piz^ /y^ X;j^) also kaowi, so 
the ’a posteriori* probabilities pCx^/y^j for i»4>^ 
are by Bayes’ rule s 


P(xj^/yj»%> 


p(x^/yj) pCzfe/Xj^^yj) 
2 p(3E^/y^) 


pix^/jp pCzfe/y^) 
I pi\/y^)p(z^yp 



» pix^ /yp 


(23) 


Iha result of leakage of information through such 
cascades of ehatnnels is generalised by proving that this 
is also valid for our concept of cf-infonaation of type I 
given by (1) l*e. 

I^djY) ^ I^(Z|2) «t> 0, oc ^ 1 (24) 

the proof of this follows from the followings 

Lemma I As *<** Information Is transmitted through cascaded 

channels from X to 1 to 2, the ®<-©qt3i vocation increases i.e. 


H^(I/Z)> H^(y/T) 


o( ^ (<=^ I (26) 


tlQStt* %(X/2)*-|JX/X) « %(I/Z)* |p(yj)lo|^ 


« ^(X/z)- — s: p(yi»«j.)log z p^{x^/yA 

!•< 3 ,k 1 * j 
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p(y4» 


2j^)iog 2 



( 23 ) 


« ^<(X/Z) - ^<2/1, Z) ^ 0 

jW (77) of chapter 2 

Q« £# 

Broof. of (84)1 Dslug (BS) 

I^<XiT) » H^(X) - H^(XA) » H^(X) - H^(J/2) « I^(ZiZ) 

Q« £• 

For oC = 1 the result (24) is proved on page llS^Ahramson (1963). 
The equality in (26) and hence in (24) will occur under the 
Bm& conditions as for • 1} this case of equality for «< » l 
is also dealt with in the sane reference. 


6. BEHAVIOUR AID ^-CAPACITIES QF SPECTHC GHAMBLS 
6.1 noiseless Channels 

We know that in such chaonels every letter of the iz:put 
alphabet is in l-l correspondence with a letter of the output 
alphabet. So Jjoint probability matrix is of the form 


P(X,t) » 




P ( x 2 , yg )....0 
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and channel prohaMllty matrix and conditional probaMlity 
matrices are 

""l 0 — 0 


P(Y/X) a P<X/Y)= 


0 1 - 


0 0 




(i) 

[^(X,Y) a H^U) a ^(Y) a (Vl-«<) lOg S 7 ^) 


H^(Y/1) a \il/r) « 0 
(ii) Capacitjea 


(1) 

^ ri 

» max 

- :^(X/Y)1 

1 

(2) 

L— — ^ 

r n 

a max 

^^(Y) - ]^(VX) 


a Max B^a) a log n 
a log n 


(iii) Bednndancies 

1 

(A.hs.Bedundancy) a log n - i^CX) 

«< ^ 

2 1 

(Abs. Redundancy) “ log n - I|^CY) = (Abs. Redundancy) 

1 

CHel, Redundancy )^a i - (I|^(X)/logn ) 

2 1 

(Bel, Redundancy) « 1 - (l^(Y)/log n) a (Rel, Redundancy) 
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^^-Efficiencies 

1 

(Efficiency)^ * (^^(X)/log n) 

2 1 

(Efficiency) a (^^(Y)/log n) =* (Efficiency) 

Cl( ^ 

In fact m liave 

12 1 
(Efficiency) * (Efficiency) a l - (Bel. Redundancy) 

oC o< ^ od 

2 

» (1 - Rel. Redundancy) 

Uv) Characteristics 

i) The “(-uncertainties both at the sending and 
receiving ends are the same. 

ii) Both the conditional “(-entropy and “(-equivocation 
are zero. This ©nphasizes a nonambiguous transmission Irres- 
pective of measuring information by any of the “(-entropies 

«( > 0. 

iii) Both the types of “(-capacities* “(-redundancies, 
absolute and relative, and “(-efficiencies are same. 

6 . 2 . sEjmm&jsLm iMDmmmm irput - ou-iput 

It will be shomi that such a system does not transanit 
“(-information of either type, < >0, i.e. Ia((X*,Y) aIc^(yjX)*sO 
and hence “<- edacities of both types are zero. 

He know that for such channels an input letter Xj can 
be received as any one of the letters y^ of the receiving 
alphabet with equal probability. Bo 
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P(X,T) 


n m 
S 2 I 
1=1 J|»l 


'“l 

^ 1 * Pi 

^2 

^2 P 2 

Pn 

V*‘ Pn^ 

n 

n 

2 

a Pi SB X 

1*1 

iSBl 


yj) - Pij « Pj 

pC:^) s m P]^ and p(yj)= i 
P<%) P^y^) » Pj^ « p(x 3 ^,y^) 
P<x^/yj) » ap^ and pCy^/x^^) » i/a 


^ *1 ^ T* * 1 ?^/ \ ^ 


H^(x,l) « — log 2 

q( « "I ^4 


(log a ♦ log S ) 

l-«c 1 i 

... (26) 


H^<X) « log S (a p ) SB _ . log a -i— log z pf (27) 

Xi **** ®C JL ^ **► ^ ^ X 

1 «< 1 oC 

u ------- log X ( S p^) se log X (l/a) * log a (28) 

1 **^ J 1 1 “ 0 ( J 


UXX/t) S p(y*)log X p'^(%/y^) 


* ®c «c 

log 2 m pr 

l-«< I ^ 


H^(X) 


« * • (22) 


H (Vx) » 3;^r i 1 


« &y (27)1 
S p(x ) log 2 p (y.^. ) » lo 


» log a » (30) 

[W (28^ 
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The etat^ent made in the beginning of this section 
follows from by (29 > and (30). 

Hence we can conclude that if wo use «<-entropies to 
stndy such channels* they retain their special feature of 
having the largest internal loss in contrast to noise- free 
chann<^s which have the least. 


It will be shown that for such channels the conditional 

«c -entropy «<> 0 is independent of input probabilities and 

( 2 ) 

hence C_, is the same for all =<>0 but, unlike as for * 1, 
I^(XiY), for > 0, «C 4 1> is not maximui for equiprobable 

even when all the. p(yj} are equal. 

l^nce in such channels each ix^ut letter in transfomed 
into a finite number of output letters with a similar set of 
probabilities for all the input probabilities so the channel 
matrix P(I/X) contains identical rows and Identical columns 
but not necessauEfily in the same position. Consequently condi- 
tional *<^entropy pertinent to the letter will be 


H-(Y/ix-) * — i— log 2 P (y./*. ) • constant » h (say) 


1 • « 


r 1 


ior o>-W- 


Hence 




m X pC^. ) ) * h 

1*1 i i 


^ 0 * «( 4 1 • 


The result is known to be true for * 1* 




Max j^(Y) - H^(Y/X) » log m - h 
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If I^jClsZ) ^ then all pCTj) are equal and tbis 

implies tbat in tMs case all p(x^) will also be equal and 
matrlK: fil/T) will also have Idaitical rows. So in this case 
IjvCXjy) » log n - b* where h* * ^(Z/y. ) for all y.. But in 

•* ^ (l) 

this case log n • h* will not b© in general equal to • 

the following exaieple would show this s 
let the channel matrix b© 


2/3 

1/3 


1/S 

2/3 


and pCx ) * 11/20 , p(Xg) » 9/20 


Considering the base of the logs to be lO we shall have 


H (Z) 
1/3 


1.5 log 


1/3 

(11/20) +(9/20) 



.30046 (3i) 


H (X/t) » 1.6 (31/60) 

1/3 


1/3 1/3 

log ((22/31) +(9/31) ) 


+ (29/60) log(^(ll/29)^\( 18/29 )^^J 


2 9172 


(32) 


When pCxj^) * p(x 2 ) * 1/2 we shall have 


M iZ) * .3010 (33) 

lA 


r 1/3 

H iJ/ty M 1*6 log (2/3) + (1/3) 

1/3 

From (3l) and (32) 

I (Z) * 1 <X/I) » .00873 

i/$ lA 



.29266 


(34) 

( 36 ) 
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and fiecBBi (33) and (34) 


H (X) • H (X/t) « .0083S 
1/3 l/3 

tim result follows from (35) and (36) 


( 34 ) 


^•4 BSC 


Let p(0) « p(x2^) » r $ p(l) * • 1 * r 


p(l/l) * p(0/0) * p } p(0/l) * p(l/0) * q 


H (j) » 


1- oc 


r *<1 

log r + (1 ^r) 


oC s<. 


log (P 4 ) 


H CX/X) * 2 p(xj — Log (p'^+q*^) 
*< 1 1 1- « 


1 << 

log (/ ♦ q ) 

1 - «C 


Since ty section 2.3 of chapter 2 ^(X) is maxieuia when 

all the p(Xj^) are equalf so hy (4) 


(S) P 1 vt at 

Q m max H*(X)- r-T log (p ♦q ) 


»!• l og(p *<l^y (?7) 

1- e< 


«<> 0, ^ 1 


» Mar Jlog(p*^(x3^)4-p'^(x2^ P(yx)3^g{A3C3/y3^)+p*^(x2/y^)) 


'-p(y2)i®g(p^(3ti>^y2^*^ 
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M Hafic 


r 1 


fp(y^)iog 


P^(xi)+p‘^(3C2) 


+p(y2)log* 


p’^Cx^) + P^CSg) 






s HaX 


l-«< 


(p p(x^Kqp(x2))log 


(p‘^(x3^)+p‘^(:^|(p pCxiHa P(X2)) 


P (Xn) 'J- p^p^ (x«) J 


my ^CDovledge iki method to maximise (38) subject to 
p(x^) + PC^tg) * P^^) i a 1,2 so fau? exists. 

In passing ve may note that for such channels 

(2} 

ay 0<cC<landP>l 

since 


Hoc(P» 1*P) ^ H^(P> 1-P) > Hp(p, 1-p) 


6.6 e.j .g 

As Already knovn that for such channels li:q}ut contains 
two symbols |^0, ijand output three symbols ^0, y, ij . The 
letter y indicates that the output is erased aM no determi** 
nistic de<^8ion can be made as to whether the transmitted letter 
was 0 or i. 

Letting p<0) a r, p(l> a l-rj p(0/0> a p(i/i) a p 
p(y/l) a p<y/0) a qj p(0/i) a p(i/0) a 0 and p ♦ q a 1 
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¥e shall have 

* q ]^(X) 

I«((X}T) a Etc(X) - (1-p) ^<(X) a p 

Hence 

C«c a P Max ^<(X) a P ( 39 ) 


H^(Y) a (l/l-«()ljog (<r p)*^ + q^+d-r) p ) 


a (l/l-oC) log 


" oC of 


i(,((Y/X) a (1/ !-*=<) log (p‘^+ q*^) 
Hence 


C2) 


Max 

l-rj 


(l/l-‘<)log{(r p)*^'i'q'^*»'(l-r)%*^] 




-(l/l-< )log(p%q'^ ) 


I think that the expression (l/l-«)log( (rp)%q‘^+(i-r)'^ p*^) 
can be maximised by the technique of geometric programming, 
Duffin, Peterson and Zener (1967), 


7. V'£HIglCA3?lOMS Qg H£S11J!S 

In section 6 of chapter 2 are developed alternative 
definitions of conditional <<- entropy and =<-equl vocation. 
Rewriting them we have 

^<Y/X)a(l/l-«C)log S p(x^) 2 p'^Cy^/^^) « > 0, « f 1 (40) 


H^(2/T)*(3/l-^)log 2 PCy^) S P*^(x^/yj) 


«(> 0, « 4 1 (41) 




There It was also shown that for 0< °c< 1 B^(X/j;) given hy 

(6) of chapter 2 is less than that given hy (40) and for 
^ 1 it is more than (40). Similar interpretation can be 
had for H^(X/Y) f Hence o(-informations defined by (i) and 
(2) would be less for 0< =< < 1 and more for °t> 1 if we 
measure conditional “t-entropy and oc-equi vocation by (40) and 
(4i) in place of (€) and (6) of chapter 2. 

All the results> except those given by (32), (38) and 
(39), proved in this chapter are also true for these alternative- 
definitions. If we define Q<(y/X) and ^(2/Y) by (40) and (4l), 
(32) would becMae 

H (X/X) «. 29176 

1/3 

and the result pj^oved there is still valid. (38) would become 


( 1 ) 

» max 


" 1 
l-«< 


log 


p\x^)*p\x2^ ? 

{pp(x^)+qp(x2^ Cp^p'^(x^hqV^(x2)) ^ - 

•♦'(qp(x^)+pp(X2 ^ (<n?‘^(x^)+p‘^ P*^(X2 ) ) 


oc> 0, o( 4 1 ...*(42) 


But the evaluation of (42) is as difficult as that of (26). 
(39) woifLd becoae 


( 1 ) 

a 1 « max 


1- o( 


of 

P ^ { 3 ? 4- (1-1^) ^ 

“C > 0, ot 4 1 


The expression in bracket too, I think can be maximised by 
the technique of gecMmetric programming. 
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Ibe atove three violations involve coffiputations. 
So on the whole all the main theoretic resnlts proved 
in this chapter are true for the alternative definitions 
given hy (40) and (4l), 



CHAPTEB - 4 



Shannon (l948) gave two coding theor©Qs for discrete 
memoryless channels. These theorems led to the development of 
a very useful application of information theory viz. The 
Coding Theory. In this chapter we shall generalise these 
theorems with the help of our concepts of “(-informations and 
“(-capacities. Since these generalisations will tend in the 
limit, as ®( 1, to the present available statements of these 
theorems, so in a way our efforts would be directed to embed 
these theorems with a dynamic character. 

In section 2 a generalisation of Shannon's fundamental 
theorem for noisy discrete memoryless channel* is stated and 
the proof is given for binary symmetric channels. In section 3 
Fano bound is generalised and on its basis the converse of 
fifliannon’ s fundamental theorem for noisy discrete m^oryless 
channels is generalised for a family of extaasions of such a 
channel* In section 4 a generalisation, based on the genera- 
lisation of Shannon’s theorem for noiseless coding given in 
Abr^son (1963), of Campbell (1965) coding theory is given. 
Since Campbell coding theory is a generalisation of Shannon's 


theorem and our generalisation is a generalisation of Campbell's 
theorej^. In section 6 talcing the usual average length I.» 2 p(x 3 L)n^ 
associated with the transmission of a set of uniquely 


decipherable code words of lengths ^n^, ng, 


with 
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probabilities |p(Xj^), , p(x^)] i bounds fori, in 

terms of Sense's entropy are fonnd. As conseq.uences of 
these '^“bomds the requiraaent of successive equip robable 
partitioning for the optimality of Shsinon-Fano code and 
the sufficient condition of the knom result for X attaining 
the lower bound ^^len entropy Is measured by Shannon’s 
definition are proved* scaae problems that these bounds give 
birth to are also given. 



By (2) of chsq>ter 3 ** ?[;^(y/X^ is the transferred 

«<-information of type II and may be termed as rate of order «< 

(in abbreviation ’’“C-rate” ) *<>0, «< 4^ 1 at which “(-Informa- 

tion of type II per symbol is transferred through the channel. 
Since the base of logarithm has thro ’out been taken to be 2, 
so ' 'bit** may again be t£^en as the unit of Its measure. The 
maximum of this «(-rate has been defined as ^-capacity of type II. 

Similar interpretation may be had for the expression 



It is possible to find an approp^ate encoding procedure 
to encode the output of s» a discrete memoryless source, so 
that the encoded output can be transmitted through a noisy 
channel at an “(-rate [^(X) - ^<(y/X^ < 0, « 4 i 

(o< is fixed for a particular ojnsideratlon ) and decoded with 
as small a probability of error as desired. 

In the limit, as 1, the above statmuent tends to an 


already available statement. 
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Let fx? 

L > 





be a ffieseage ensemble 


of the sooaroe s 

(1) the H messmgee are seXeeteS tdtb equal probability 
i*e») the seXeetlon is maeie oompleteXy ut raadcmi 
Cii) tbe obannel is speoified by 


p(0/0) m p(l/i)wi p, p(0/l) » pCl/0) « q, q< l/S and p^q * 1 

(lii) all the 0a.&»4^ messages have the same length of 
n Mnaj^ digits} 

(iv) the binary enio^ing procedore is a ramify one* 

Mm^m m49^m:§ 

W» have II souree messages each encoded with a binary 
digit®. At the receiver m have a catalog containing all 
possible 8*^ n»3ymlol ssquqnces «ith i source sequences 
i^eciflcally underlined* In case the received message does 
not agree ulth any of the II source ssquencesj ue take that 
source sequenee as being transmitted which differs from the 
received sequence in least no. of digits. 

A sequence of n i*8 and 0«s can be considered as a 
sequence of n indep^ent Bernoizlli trials in which the 
correct transmission of each binary digit occurs witb 
probability p and erroneous transmission with probability 
q • 1 • p < 1/2 • falling Z as a raadas variable to represent 
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the no. of erroneous digits in a received message, ve have 


so 


n n-k k 

P(2 * k) » ( ) p Cl 

n-k 


n 

E(a) « S 
kaO 


n 

k( ) 
n * k 


n-k k 

p q • nq 


fhus we expect, on ‘Uae average, in a sequence nq 
digits to be altered by noise, therefore, according to our 
decoding procedure, frcsm ttoe list of underlined sequences 
representing N source sequences at the receiving end each 
sequence that differs from the received sequence by nq or 
less di^ts could be expected to be the transmitted sequmce. 
so the expected no. of sequences that can be considered as 
possible original message for a tran^itted sequence is 

nq n 

M » 2 ( ) (1) 

kaO k 

For q < 1/2, (i) Can be written as 

n 

M <1 ♦ nq ( ) (2) 

^ •n<v 

But for large n w# have by strling* s foapmula 

•n jjA 1 

nl ^ /2r e n 2 (3) 


Using (3), (2) reduces to 



76 - 


**1IP 

M ^ 1, ♦ / (nq)/2irp p q ^ (4) 

|Wr details of this step[ B&^a(196l^ 

of these 14 sequences* only <me is correct and M-1 are 
potential misinterpretations of th© transmitted signal* 

By assiaaption (iv) the probability of selecting an 
n-digit sequence so as to correspond it to one of the H 
messages of is H/2*^ j this in turn reduces the 

expected no. of misinterpretations of the recei’^d sequence 
for a transmitted sequence corresponding to one of the H 
messages of from M-i to (H/2°)(M-1). So denoting the 
frequency of the occurrence of an error by we have 


^ iM/^) (M-l) 

Using (4), (5) can be written as 


M . 4 (li/2*^) /■<nq)/2r^p 


We know by (37) of chapter 3 that for a BSC, 


(2} 

* 1 • (l/l-*<) Io^Cp”^ *► q**^) 

(S) (2) 

or 1 * (p“Vq"^)*^ (2’^®^ ^ 

2 

( 2 ) ( 2 ) 

or (1/8) - (p^+ q*) ( a ■< ) 


(S) 


( 6 ) 


( 7 ) 
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Using (7), (6) can Ue written as 


(2) (2) 




1> -np -n«5p 

P q 


<K a( n 

(p + q > 


/ nq 
2 w p 


( 8 ) 


■S,a,se.,.(i) ,,yte, l 

Now by suitably choosing N, the no, of messages of^X j 

C2) 


to be equal to or less than 


2 *^ << / n 


, <8) reduces to 


n 




(p%q ) 2 


.< X 


q 

2nir p 


Now we shall prove that 

-p -q 

p q 

(P°‘+4‘) 3"‘0- 

(10) can be written as 


( 2 ) 

o( c<. ®^Cl*C^ ) # / P a\ 

(p + q ) 2 ^ (1/ p^ q^> 


(9) 


( 10 ) 


or log (p% q®^) + <=((1 - • log (P** q**^) 


( 2 ) 


( 2 ) 

or Cl-«C) ^(Pfq) + «<(1- ) > Hj^(p,q) 


( 11 ) 


But from (37) of Cha 4 »ter 3 we have, in particular for 

0 < »c < 1 , 
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Mow ve shall prove 


-p -q 

P q 


OC oc (rC®) i2} n 

(p ♦q ) -•<(<«( -D- ^ 


( 2 ) 


^ 1 




a 


(13) cah he written as 


<sC ^ (2) P Q 

log<p ^ ** 1) * P ** logCP q ) 

or 

(2) ( 2 ) (2) p 

(!--=<) (1- )‘t Cb( • oCCCef -l)- P > “logCp q^) 

or 

( 2 ) ( 2 ) ( 2 ) 

1- - «C+ oC Qj ♦ - «c +«< - ?> 1 - 

or 1 • ^ ^ 1 • 

or **1 


Conversely hy onr st^posltion 


(13) 


M 0^ 

or 1 - ^ ^ 1 - 

and further retracing the steps we shall have (13). 

So (13) is establishedi hence from (12) *** 0 as n-* ®o 

irrespective of the noise characteristic (q < 1/2). 

This ccMipletes the proof. 
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3. omeralisahon of bdmd amp ths converse of 
3*X A.Qsm ralisatlQa of Fang Bound 


Fa»o bound gives upper bound for equivocation in terms 
of the proi>ability of error page 163, Abrsason (1963^ • 

Below this lx)uad will be generalised i.e. , an upper bouisi, which 
tends to the known result, as 1, \dll be established for 
°C- equivocation, < ^ 0, *<4 1 • 

Suppose a decision rule ® chann^ with input 


[x] * and output {j] « [ y^, . . . . tyja”] 

assigns to "that (say) x* for which 

p<xVy^) :^P(3c^/y^) ^03^ all i. 


be such that 


(14) 


We know that this decision rule Is sometimes called a condi- 
tional maximum likelihood decision rule. By Bayes* law,( 14 ) 
can be written as 


p(y/x*) p(x^) 


pTyJT 




p(yj/xi) p(x^) 
p(yj) 


for all i • 


When the a priori probabilities are all equal, the above rule 
can be written as 


d(yj) a X 

subject to 

p(yj/x*) ^ p(yj/xj^) for all i. 

We know the decision rule so defined is known as the mtaxlmum 
likelihood decision rule. 

iSlil 


MN 
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The error probatolllty Pjg using any given decision 
rule is given by 

!j 1 

Pg - z j^P<yj) [l - p(d(yjVyj)]J (16) 

(15) can alternatively be written as 


Pg » I p(y) - ^ p(d(y)/y) p(y) 


• 1 • S p(d(y), y) 

Y 

Writing pg * 1- Pg we have 

Pg * I pCx*, y) 

and 

Pg a S . P(x,y) 
Case (i) when 0< ^ < i 


(16) 


(17) 


By (7) we have 
1 




* 1 — Z Y I ^ 


But by Jensen's inequality 


p(y)log Z p (x/y) ^ log 
X 


2 p(y) 1 2 e p*^(x/y)+ p^CxVy)]! 

i IX-x J 


* log 


2 ^(y)p’'(x/y)-i* |p(y)p*^(xVy) 


Y,X-X 


M Z tf‘‘Cy)p«'(x/y)pl-^(y)+5 P°‘iy)P°‘(*Vy)pl^) 
1»X*X X J 



KM 
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Holder Inecimality) 


-- cf. l-o( ^ ^ 

=* log "JS *P(3C|y)j (n-l) +?Sp(x ,y)J (18) 

Multiplying (l8) by Cl/l-«) , a positive nmberi and using 
(16) and (17) we have 


r 0( ^ -r-» 

H (x/z) 4 (l/l-«<) log I P (n-l) + p , (19) 

S B ^ 

0< oC < 1 


SlMM -1 

Sinee Z p (x/y) ^ 1} so 
X 

o( c< 

p(y)log S p (x/y)> S p(y) S p (x/y) log 2 p (x/y) 
X XX X 


> 


2p(y) Z p^(x/y)|log sSp(y) z p‘^(x/y)j 

XX J jjr X -J 


(20) 


(By Jensen* s inequalliar) 

Btt I p<y) 2 p*^(x/y) ^ 1, let it be p j so using this fact 


(20) reduces to 
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2p(y)log zp (x/y) > P log 
I X 


I p<y) s P (x/y) 

X X 


plog 


1 


2 ♦P(y)p‘^(x/y)+2p'^CxVy)pCy)j 
X,X-x Y J 

I ■ 1*^ p( ^ IhpC 

* ^log [^2 *p‘^(x,y)p (yH Sp (x^,y)p (y) 


l-o< 


^ ^ +{2p(x ,y).? i (2i) 

( ^ oUeA. 

MiJltiplylng ( 21 ) by (l/l-«c) , a negatdLv© number, 
using (16) and (17) and tb® fact that p < 1 , ve have 


H (XA) < ( 3 /l-cc) log 


, . l-«( _*0< - 

Pg (n-i) + p 

£ J 

^ > If ^ 1 


(22) 


So combining (19) and (22) we have 


(l/l-() log rp^{n-l)^'^+ Pg"*! 

0, o£ 4 1. 

In the limit, as «< -* l, (23) tends to 


( 23 ) 


H^(X>^) ^ H^(Pg) 4. Pj. log (n-1) 


as already known 
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3.2 A Generalisation of fhe Conver se Of Shsoanon^s Second 
Emdactental Theorem 

A tom of the converse of ^aannon’ s 2nd fundamental 
theorem as applicable bo '=<-entropies is stated and the proof 
is given for a class of channels - a family of extensions of 
any discrete menoryless chanml - and when the input proba- 
bilities are equal. For this we need the following l<aftma. 
Lemma ; ‘<-capacity of either type of an nth order extension 
of a discrete memoryless channel is n times the «=<-capacity of 
the corresponding type of the channel. 

We know that an nth order extension of a discrete 
m^oryless channel with input output 

\y| « [^1*^2’*****^] ^ n-ttyples of the form 

(x^,X 2 ,. .. . * u (say) as Its input and all n- tuple of 

the form * v (say) as its output and 

p(u) a p(x^) .... P(Xj^) 
p(v) = p(y3^) »....p(yj^) 
p(v/u) » p(yj^/x 3 ^)....p(yjj/Xj^) 
p(u/v) » p(x^/y^) ....p(x^/y^). 

«• infoimation of type I for such an extension is given 


I^(Xr5 V) » ^(U) - ^(II/V) 


(24) 
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where U is the set of bXI n- tuples u and V tlae set of 
all n- tuples ¥, Bat 

H^(U) * (1/1-^) iog S p‘^(u) 

* (l/l*«<)log S p*^<x p*^(x^) 

U 1 “ 

» n H XX) (25) 

Ho{(U/V) * (l/l-<<) 2 p(v) log 2 p^(\j/v) 

* (!/!-=<) 2 p(v) log S p'^Cx-Zy,). ..p*^(x /y ) 

V U 1 X n a 

»(n/l«o() 2 p(v) log 2 p’^Cx^/yj) 

V" X 

* a ^ (2/1) (26) 

Using (25) and (26), (24) reduces to 

I«((Uj V) » a I^(Xi I) (27) 

oc- capacity of type I is given ^ 

Max. I (U} ¥) 

U *< 

so frcBi this and (27) the statecient of the l^uma for this 
part follows. 

Similarly we can prove that 
!«((¥; U) # n I^(Y-, X) 


( 28 ) 



• 86 ” 


Since ®<-capacity of type II Ig given by 


Max. I^(V 5 IT) 

so from this and (28) the result follows for '^-capacity of 
type II. 


Given 6> 0, it is impossible to ^code a set of 
n(C^l^ + e) 

messages in 2 *< code wards so as to be transmitted 

through an nth extension of a noisy discrete mmoryless 

(l) 

channel having *<-capacity with as eaall a probability 

of error as desired. 

As 1, this tends to an already available statement 
for such a family of extensions. 

Proofs Suppose the nth ertaision, of the type ccaisldered 
in the lemmai of the channel has ♦£) * M (say) code 

wordst each to be transmitted with probability l/N . so by 
the above lemma and (S3) for 0 < «< < i we have 


^^(U)- H (B/Y) > log 


||<(H-l)i“^+ a/(i-«c) 


/ 1 & ) ^ 


(l/l-oC) 


1««0( <*o( 

i*ir ^ pg 


n 1“^ < 1*^ -«c 

iM/2 ^ 


( 29 ) 
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putting K 
reduces to 


* 2 


( 1 ) 


to the left side of (29), it 


ne l-«c 
(2 ) 


0 ( 




(H-O + 


Cl - Ps 


(30) 


For any «< from 0 < «< < 1 , the right side of (30) 
is 1 iff Pjg * 0 otherwise it is always more than Ij 

but the left side of (30) is always more than 1 and cannot 
be made equal to 1 howsoever small n and 8 we may taJcej 
actually for a fixed g, the more the value of n take, th© 
more away will Pg be from 0. 

Again by the above lemma and (23) for 1, we have 


(l) 

n 

2 ^ 

n 




oC l-c< o( 

Pg(S-l) +(1-Pg) 


-J 

1- «c 


( 


( 1 ) 


h c;^ 

2 ^ 

~ ) 


oC-1 


> 


Pg(N-l) 


1*^ 

+ (1 



1 c< !•*< oC 

( ) ^ Ps CH-1) ■¥ (1 “ Pe ) (31) 

2^ 


For any from z' 1, the right side of (3i) is 
1 iff Pg * 0 otherwise it is always less than 1, but the 
left side of (3i) is always less than 1 and the more the 
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n the smaller the left side ahd for the inequality (31) 

o( 

to hold the tewa (l - Pg) on its right side should he 
made comparatively small and this requires Pg to become more 
and more i*e., to deviate away and away from 0. 

This cample tes the proof. 


4. 


For a uniquely decipherable code with word length 
given by the sequence ^n^| and transmitted through a noise- 
less channel with probabilities giv©i respectively by the 
sequence (p(Xj|^)| » Campbell (1965) defined a cost fvmction 
of order t, 0 < t < i 


Kt) * 


(1/t) logjj 


/ V ^ % 

2 p(x^) D ^ 
i 


(32) 


where (l/B-t) « «< and D is the nuaber of letters in the 
encoding alphabet* and proved a coding theorem for noiseless 
channels when the entropy of the source is measured by 


i|^(X) * (1/1-^) logjj 2 «<> 0, °c 1 (33) 

The re stilts he obtained tend in the limit* as *=< *♦ 1, to the 
already available results of the corresponding Shannon's 
first coding theor^a for such channels. 

A generalisation of Shannon's first theorem is given 
in Atramson (1963)* There it has been argued that since for 
a received symbol jj we can associate the a posteriori 
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probabilities pCxj^/y^) with the input alphabet? so keeping 
in view the interpretation of shannon* s first theoreo for the 
entropy of an alphabet as the average nmber of D-ary letters 
necessary to represent one symbol of the alphabet, the average 
of the a posteriori entropy H^(y/y^) over jj may likewise 
be interpret^i . fhis idea gave rise to that generalisation# 

fhe difference between Shannon's first theory and that 
gmeralisation lies in the fact that the former deals with a 
fixed set of input probabilities whereas for the l^^tter the 
set of input probabilities changes with the receipt of every 
output symbol# the knowledge of every received y^ gives rise 
to constructing a D-ary code for the iz^ut alphabet; so if the 
output consists of s symbols we have to construct s D-ary 
codes and use the 3 th code to encode a symbol frcaa the input 
if the received symbol is yj • this tantamounts to choosing 
a sequence of code words from the known sequence of s codes# 

For this purpose all these s codes should be instantaneous 
because if they are tsdcen to be uniquely decipherable codes 
then it is not known that the above choice of code words from 
these 8 uniquely decipherable codes will result in a uniquely 
decipherable code# So that generalisation is true only for 
instantaneous codes# 

In this section we aim at proving a same sort of genera- 
lisation taking the Benyl entropy given by (33) as the basic 
measure of uncertainty# As pointed out in the section 1 that 
our generalisation is a generalisation of Campbell coding 
theorem and since Campbell coding theorem is a generalisation 
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of Siiamion»s first fundamental theorem so our generalisation 
is also a generalisation of Shannon’s first fundamental theorem. 
foT this we need the concept of ®{*equivocation| <<^0, 

4 1 > associated with a chama^ when the channel input 
entropy is measured by (33). 

By (6) of chapter 2 this is given by 

H^(X/Y) * <3/l-«) S p(y^) log % /y > (34) 

j i • 

We also know from the proof of l^ama of section 3.2 that for an 
nth extension of a channel with input probability distribution 
as a product distribution 

( 35 ) 

Campbell (19^5' ) also showed that 
C3/t)3cg|)S p(x^) Cl/l'-=<) log I p‘^(x^) (36) 

where (l/l^t) * «< . 

As in the generalisation of shannon’s first theorem 
referred to in the above paragraph and on the lines of which 
our generalisation for Campbell coding theorem is based, we 
shall consider s B-aury codes corresponding to s outputs. We 
shall assume that these s codes are instantaneous and their 
word lengths are given by 
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Table 1 

Input 

Code 

Code 

Code 

symbols 

r . 

2 


^1 

“ll 

^12"“ 

^-^is 

^8 

MV 

022** 

ngg 

X 

r 

“rl 

^2-“ 




tlae pro lability distribution attached to the jth 


code is 


r 


p(x^/yp. 




PCXy 


So for the itii code* (3€) reduces to 


tnii of 

(i/t)iog z p(Xj^/y^) B ^ (i/i’<)iog z p 


i 


i.e. L (t) > ^<(1/ y^) 


Denoting the cost of transmitting an input symbol by 
} we have frcaa (37) , 


(37) 


( 1 ) 3 

L (t) * Zp(y.)I.(t)^ ^<(2/r) 0< t < CO 

3 


(38) 


As laid 4own in table 1 above, n^^ U the Imgth of 
code word for the i%ut symbol when yj is the output, 
so choosing so that 
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«c log pCx^/y^) log S p“^(x^/y^) ^ 


( 39 ) 


<- ‘=<log pCx^/y^) ^Jog z p^(Xj^/yj>fl 


where base of the logs is taken to be D* (39) can be written 


3»og P^i^/7^) Z p’^(X|^/y^) 4 : log log D p (x^/y^ )Sp*^(Xi/y O 

i i 

or 

-cC t 

p'^CXi/y^) s ^ p’^\3j^/yj|jp*^(x3^/y^^40) 

Multiplying (40) by p(Xj|^/y^), suoaming over the subscript i 
and Using the fact that (l/l+t) * «< we have 

(2 P^(Xj^/yj)) ^2p(Xj^/yj)0 ^ < 0 ^(s p^(x^/rp) (4i) 


In (4i), first taking logarithms to the base B and then dividing 
by t we have, 


i 

H^(X/y^) 4 L it) < (2/y^) *► 1 (42) 

Multiplying (42) by p(yp and summing over i we have 

(1) 

I^(3E/y) ^ L it) < \ (X/X) ♦ 1 

Similarly for an nth extension defined in section 3#2 we have 


( 43 ) 
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Using (36), (4S) redness to 

Cl) 

Ct) 1 

UjlS/t) S \ CX/I) + ^ (44) 

fhe attainment of Uound in (38) follows fro® talcing limit 
of (44) as n ■* • 

If t » 0 tbis is inst tlP goner all cation given in Atraiiison 
(1963). If t « we cboose eacla n^^ for all i to satisfy 


log ? 4 : < log r ^1 


(46) 


i i 

Since It I (t) * L (®^ » max n. . , so (46) can be written as 

QO i 


log r X I (eo) < log r + 1 


(46) 


Multiplying (46) by p(y^) and summing over 5 wt bave 


^(3C/Y) 4 I. < I^(X/Y) *► 1 


where is the value of ^(X/I) at «< » 0, 

Similarly for an nih. extension we shall have 


Ho(lf^/I^) 4 : 1 ^ ♦ 1 (47) 

Using (36), (47) reduces to 

HgCx/i) < ifeCa'i) + i/„ («) 


the resiilt follows trm (48) in the limit as n °® 



This completes the description of a coding procedure for 
attaining the bound in (38), 

5. 

5.1 o(- Bound a 

let 

L a 2 
i 


where S p<x^) *l and p(x^) is the probability of transmitting 
through a noiseless channel from a uniquely decipherable code 
a code word of length n^. for the ith input symbol. Shannon 
gave the following lower bound for I* 


L > 

log D 

where H^(X) is the Shannon* s measure of entropy and D is 
the no. of letters in the mcoding alphabet. The following 
theorem generalises this bound. 



Th. 1 Let X a 


j 


1 » ^ 2 *****» 


XjpJ be a message ens^ble 


from a memoryless source with respective probabilities of 

r 


transmission given by 




so that^S p(x^)al. 
If X is encoded in a sequence of uniquely decipherable words 


of lengths p^jng, , . . . ,ny 
letters then 


from an encoding alphabet of D 


__ r 

^ °1 ^ 


(X) 


(i) 


log Q 


« > 1 


(60) 
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-ni 

(il) If in parUcular pCx^) *0 /s D theiip< «c< 1 

^ - (log 2 1) )Aog D 

log D i 


( 61 ) 


£l.02l« (i) 

By Jensen* s inequality 


o(-l 


‘<ri 


log 2 p^(x^)= log J p(3t^)p (x^):^ J p(Xi)log p (Xj^) (62) 


We know that if jpCx^)J and [qj are two sets of non- 
negative numbers such that ^ p(xj^) * 2 q^ * 1 then 


- S P(3c^) log pUj^) -5^-2 P(x^) log q^ 


(63) 


For «c ^ 1, (6S) can be written as 


2 p(x^) log 2 p(Xj^) log q^ 


*<-1 


(54) 


Putting q, « B ^/2 B in (54) and using (62) we shall 

^ X 


have 


log 2 p‘^(x 4 ) L(l-oC) log D - (=C-l)log 2 D ^ (65) 

i ^ i 


Dividing (55) by (l/l-«() t a negatd.ve no., and using the 


- n . 


result of HcMillan* s theorem that 2 D ^ ^ 1 for a 

uniquely decipherable code we shall have 


1 > ^<(X) / log D 


'<> 1 » 


( 66 ) 
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Since this result is already known to he true for «t =s 1* 
so combining this with (56) we shall have (60), 

It may be noted that this result also follows from 
the known result and the monotonic decreasing character of 
Boc(X), «< > 0, proved in section 2.2 of chapter 2, The above 
is an independent proof. 


(ii) 

Since p(x^) 




so 


log 2 p’^Cx, ) 
i 1 


log Z pix^) 



*<-1 


= log 2 p(Xj^)D 




”(«C-l)log 2 
i 


_“n< 


D 


( 67 ) 


But 1:^ Jensen’s inequality 


-rn(«t-l) -n,('=<-l) 

log 2 p(Xj^)D ^ ^2 p(3^) log D ^ 


» - (o(-l) L log B 


( 68 ) 


Using (68), is?) reduces to 


log 2 p'^(x )> -(«C-1) L log D -(«-l)log 2 (69) 

i i 


Multiplying (59) by (!/!-*<)» a positive number, we shall 
have (6i). 


Q«£. B* 
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5,2 Results th at FoIIqm 

(i) It is mentioned on page 140, Reza (196i) that 
' 'If the reqiiLr^saent of successive equiprobahle parti- 
tioning of the prohahility matrix of a message «isemble 
is approximately satisfied then reasonably efficient 
encoding procedures can be expected.'* 

The proof of this statement, when pCx^ ) » D ^/Z D 
follows from the above theorem. 

Proof , When 

p(Xj^) = i^/ ? d"“^ (60) 

from (SO) and (^>l) we have in the limit as 1 


H (X) 

-i < L < 

V X 


%(X) 


log D 


log S D 1 
log D 


(61) 



that if XL subject to (60) are chosen 
*ni 

2 D '*■ beccmes almost or actually 

i 

cistence of a uniquely decipherable 
rhich the average word length L would 
equal to 1^(X)/ log D, The nearer 
■ 1 we are in respect of our choice 
) the better would be the attainment 
cing the average word length achieve 
/ log D and better the approximation 
the requirement of successive equi- 
f the probability matrix being met 


len efficient code. 


a. £. D. 
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(ii) *<wbat set of source probabilities for a luaiquely 
decipherable code would achieve the lower bound H2^(X)/UogD 
for L in noiseless coding ?** was one time a baffling 
question. How its complete solution is given by the theorem 
' * The necessary and sufficient condition for 1 to be 
equal to EjiX) / log D is that pCxj^) * D ^ . ” 

The sufficient condition of this theorem also follows 
from the above theorem when pCx^ ) ** B D . 

Proof . I^n p(Xj^) ss B B » both the results (60) 

and (6i) of the above theorem are valid . (6i) gives a 

series of upper bounds and in the limit as 1 we shall 

have the least i;^per bound for L and this is given by, 
denoting it by l.u, b, Z » 


l.u,b. L s 


%(X) 

log D 


log S 
log D 


(62) 


and for «< s i the known greatest lower bound for I» , denoting 
it by g.l.b, L , is 

%(X) 

g.l.b. 1 * ^ (63) 

log B 


From (62) and (63) it immediately follows that 


l.u.b. Xi ^ 
Hence from p(x^) * 
* D ^ 


g.l. b. Ii 

■^4 

B ^ / 2 D 
i 


iff I B ^ s 1 (64) 

■ni 

and (64) it follows that 
H (X) 
log B 


Q.E.B. 
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5.3 ProbleBis that Arls& 

(i) C^phell (1966) define^ a cost function of order t 

L(t) s (/t) - log S p(x. ) 0 0< t< oc^ (66) 

t i ^ 

1 

where oC and 0 is in the nmher of letters in 

1+ t 

an encoding alphabet used for constructing a miquely 
decipherable code and is the length of a code word for 
tlie source symbol to be tran fitted through a noise- 

less charnel with pro babililgr p(Xj^) and proved a coding 
theorem for noiseless channels by taking Benyi's entropy given 
by (33) as the basic measure of uncertainty. In the limit* 
as *=< *• 1* Campbell's result tends to the corresponding 
Shamon* a re suit • 

For 0< «c < 1 , multiplying (62) by (i/l-«c) , a 
positive nmber, we shall have 

VX) > 0^(2) 0< «C < 1 (66) 

Equality in (66) is possible when all the x^^ are equiprobable. 
When the cost of transmitting a code word is measured by (60), 
the Shannon's result says that by taking sufficiently large 
nth extension of the source it is possible to encode the 
symbols of the so extended source into a uniquely decipherable 
code such that the difference between Ln. /n and H2^(X) /log D 
is as small as desired. 

Campbell's result is restricted to 0< «< < 1 since the 
cost function (66) is true for 0< t < ooi.e. for 0< «< < 1 
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(by o( SB — ), By virtu© of (66) and the atove Sliarmon's 

1 t 

result it seems feasible to bave a uniquely decipherable 
code for an nth extension of the source so that the 
difference between /n and ^(X) / log D is almost 

negligible when the entropy of the source is measured by the 
corresponding member of |i|^(X) , 0 < csc < i j . 

Xh© beauty of cost function (65) lies in the fact 
that CaPpbell by considering it was not only able to give a 
lower bound for L (t) in terms of I|^(X) but also gave a 
method of choosing n^ for the pui^ose* But a method, as 
to how to choose n^^^ so that the resultant code is a uniquely 
de<d.phBrahle code for an nth extension and the difference 
b^twaen "Ln /n and ^j((X) / log D is as small as desired, 
is still to be developed. 

(ii) Since ]^(X) 4 H^(X) for *=<> 1, the result in (4) 

gives us a better lower bound of L for uniquely decipherable 
codes if we measure the entropy of the source by a member of 
"<> l] . 

For the already available lover bound 

log D 

We know that by choosing so that 

- log p(x> ) -log p(x-) 

Sn. < + 1 

log D * log D 

we have not only the guarantee of the existence of a uniquely 




decipherable «3de but can also make t achieve its lower 
bound* 

In this case not only a method of as to Sum to 
choose n^ in practice with a view to have a uniquely 
decipherable code so that the bound in (60) for 1 
is more or less or actually achieved but also the reasoning 
for visualising its very feasibility are to be developed* 

It may be noted that all the results developed in 
this chapter are true also for the alternative definitions 
of QsjCY/z) and ]B^(X/T) given respectively by (100) and 
(lOi) of chapter 8* 





This chapter coasists of two parts. Part I deals 
with bivariate continuous msmoryless channels and part II 
with multivariate continuous memoryless channels. 


PART * I 

1. li^mDUCEEOM AfiD OF REKYI * ^ 

As given in chapter 1 Benyi*s ^c-entropy for a 
complete discrete scheme with probability matrix 



i»l,3,...tn , p(Xi) ^0 


2 p(x. ) * 1 
i ^ 


is given by 


* ' ' ' lo^ 2 p (^4 ) 

0 ( 1 


( 1 ) 


Cl) can be written as 


-logg 


2 pCx^) 


« 


or 


iogg ^-1 


where 


H 


s pCxjl) p’^^^Cxj^) 


'C<-1 i 


w iw w iifb i w w ww . 




( 2 ) 
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is laiown. to be a power mean of order *=<-1 of n positive 

numbers p(x^)> p(x 2 )j pCXj^)* 

Corresponding to (2) the power mean of order *=<“1 
for a complete continuous probability scheme having the 
probability dmsity f(x) is 


00 

X f<x) f (x) dx 

.JOO 


«<-l 




(3) 


oo 

where X f(x) dx ® 1. So following the pattern of the 
«oo 

structure of (i) we may define entropy of order > 0, 
<< 4^ 1 such a scheme as 


H^(X) * -logg 


eC-1 

X f(x) f (X) 

mOO 



logo X f^(x) dx (4) 

1* «C 


Another way of arriving at (4) is as follows: 

Let a continuous scheme have fCx) as its probability 
oo 

density so that X X(x) dx » 1. If f(x) is finite a»d 

.oo 

positive everywhere in its domain of definition then the 

1 

Kolmogorov- Hagvaao generalised mean of log 2 “ 17 “^ ^ talcing 

^ X 

X * logo — — in the strictly monotonlc and continuous 
® f(x) 

parametric function 
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g^(x) » 2 


(l-=<)x 


“< > 0 , «( 1 


(S) 


is 


-1 


CK ) 


L 


J f ( lo J 


pa 


3L erf 

ss iQg £ f'^(x) dx 

1 - oC -OO 


(6) 


It may te noted that if we adopt Shannon’s convention 
that 0 X ®®ss 0 or evaluate (6) in the extended real number 
system, then (6) is also defined even when f(x) vanishes 
at the most over a set of measure zero. 

The power mean of order ^1 given by (3) has exten- 
sively been studied. The following are a few of its proper- 
ties that we need for otir subsequent developments 

(!) it is a continuous fmetions of page 13, 
Littlewood (1944), 

(ii) it is a mono tonic increasing function of *=<, 
page 144, Hardy, Littlewood and Polya (1952), 

By property (i) for (3), 5<(X) defined by (4) is also a 
continuous function of «c and by property (ii) for (3) it is 
a mono tonic decreasing function of 

It can be easily verified that in the limit, as 
(4) tends to the corresponding Shannon’s definition. 

Taking (4) as the basic measure of ac-taacertalnty, 
other «<-entropies associated with a two-dimensional conttnuous 
m«noryless channel are defined, some relations involving 
these ^-eitropies are developed a»ae--ipelatieas->^4n^ 
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said the drawbacks of 
i| 5 ((X) Used as a measure of entropy for a continuous 
schme are discussed* All the results developed below tend 
in the limit, as 1, to the corresponding available 
results* 

The main tools used are Jensen’s and Eblder’s 
inequalities for integrals* The case, whan the underlying 
schemes are independent, has been included while applying 
iRilder’s inequality and 3ii this case there willj^strict 
equality* In the case of Jensen’s Inequality, the equality 
will hold when the density functdon, of T)ih.ose convex or 
concave function the inequality deals with, is uniformly 
distributed. 

All the operations, dealt with below, are contingent 
upon the existence of underlying integrals. From section 3 
onwards, base of the logs will be assumed to be e» 

2, °(z.M3rQjAs^.AssQ<d^t£^ «ith_a Two Dimension a l Contlnuoug 

suppose a twa-dimensional continuous randcmi variable 
has f(k,y) - the ^oint probability density f(x) and f(y) 
the marginal probability densities , f(y/x) the conditional 
probability density of Y given a value x of X- and f(x/y) 
the conditional prob* density of I given a value y of Y 
so that 

OO OO OO CO 

f f f(x,y) dx dy » 1 , I fCx,y) dy * fCx),/ f(x,y)dx *f(y) 

«00 -OO -CO IfflO 
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f(3c,y) f<x,y) 

f(y/x) tt , f(x/y) * 

f(x) f(y) 

There are flTO «c-entropi€Sj *< ^ 0, «< 4= 1 » associated with 
such a variahle, 

(1) B^(Z), defined for the input X. It is given by (4). 

(ii) I^(y), defined for the output X. It can similarly 
be defined as ^(X). 

(iii) ]^{X,Y), defined for the system as a wh3le. Based 
on the procedure adopted for getting <4), it is 
given by 

1 ^ of 

H (X,y) • log X X f (x,y) dx dy (7) 

^ o( 2 -«J -OO 

(iv) Hj(Y/X), defined about tlte output Y when it is 
known that Z is transmitted and may be texmed 
as conditional “t-entropy. It may be defined as 
suehs 

Based on the procedure for getting (4) conditional 
<<-entropy calculated on the assumption that the event x of 
Z has occurred is 

1 oo 

H (I/«) = log^ ; AyA) dy •« >0, « t 1 (8) 

3to have a measure for H (Y/Z) it is auite natural to 
take the expected value of <8); so 



*■ l07 “* 


\iY/X) 


1 

l-o( 


oo 


f f(x) 

4J00 


r- 00 ^ 

log ! f'^Cy/x) dy 

2 ^00 


dx, 1 (9) 


(^) I^(X/X) , defined about the ii^jut X when it is 

known that Y is received and may be termed as 
e(* equivocation. Based on the procedure adopted 
for getting (9), it is given by 


H^(X/Y) 


1 ^ 

— ; f(y) 

«*O 0 


00 


log X f (x/y) dx 

2 **00 


dy ®£>0,=< 1 

... ( 10 ) 


In the limit, as “C •* 1, (7), (9) and (lO) tend to the 
corresponding SSiannon*s definitions. 

Since 


so 


Lt 

o( 00 


OC 


/f (X) ax 


»co 


^-1 


a max. f(x) 


j^age 143 Hardy, Littlswood and Polya (1962)^ 


(i ) lit £|g^(X) ® •lo^g max. f(x) 

<ii) Lt H (X,Y) * •log^ max, f(x,y) 

af., 00 o( 


aC ^ 00 


00 

(lii) Lt HXX/X) » - X fCx)/log^ ffl^,fCy/x)) dx 

^ „O0 I » IT / 


L 


) 

--J 
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3* ReXaU ons Between J<-i3atropl&5, 

3.1 Belation Mtifeen i|<(I/X) 3SA.\(J) 
Case when 0 < «C < 1 

Since log z Is concave, so talcing 


oo 


X « X f (y/z)dy in 




oo p oo <5( 

/ f(z) log X (y/x) dy 

oo -oo ^ 


dz 


and applying Jensen's inequality for integrals Page l5l, 
Hardy, littlewood and polya (1952), we have 


OO 1 

oo 

1 


OO 

~ ®°c< 

X nx) 

log X 

<0^ 

f (yA)dy 

1 dx 

^ log X f(3c) 

1 f Cy/x)dy 

-OO 

-oo 




.JOO ^ 


oo oo J^-c< c< c( 

a log ! t t (x) f (x)f (y/x)dx; dy 

-oo mX)0 


log 


«< 

/ f ir) 

-oo 


oo 


**1 «iiO|[ 

X f (x/7)f (x)dx dy 

m/OO ^ 


(11) 


But in 

OO e< l-oC 

X f <x/y) t (x) dz 

«iCX) 

taking k * (!/'<) > 1 and k* « (l/2r*<) so that (2/k)+(3/k*) = 1 
aiid applying Holder's inequalli^ for integrals for these value 
of the parapseter k, page 140 Hardy, Ltttiewood and Polya (1962), 
ve have 





^ oc 1-*®C 

S t (x/y) f (x) 6x < 

mOO ^ 


®( 1-sC 

- oo 1 r oo "1 

X f(x/y)dx /f(x) dx/ * 1 
-oo J -oo J 

j_ 


oo 


c( 1“°C 

life max. X f (x/y) f (x) dx = p ^ 1 . By tMs (11) 
y -oo 


reduces to 


oo 

X f(x) 

«oo 


oo 


oo 


iog / f (y/x) dy dx 4 : log p + log X f <y)<iy 

«iOO ^OD 


Multiplying this ly (Vl-«C) , a positive no,, and using 

tlio fsict# P ^ 1 w*© h-Q,V0 


(T/X) ^ ^^iY) 


0 < « < 1 


( 12 ) 


Case uhen «( > 0 


00 


Since 


so 

00 

; fu) 

«»00 


X f (y/x) dy < 
- 00 ^ 


' 00 


X f (y/x) dy 

-OO 


* 1 > 


(13) 


OO. 


[log X f (y/x)dy 
-00 


OO 


oo 


00 






• • • # 4 14 ) 


00 


Since x log x U convex, so taking x » If (y/x)dy on the 

woo 

ReHeSs Of Cl4) and applying Jensen« s inequality for integralst 

we have 


RelmSe Of (l4} 


OO 


00. 


IJ(x)|l^f (y/x)dyjdx| log 


00 


00 


IJ U)[IJ (yA)dyl<u 


» # « 


( 16 ) 



> 


so taking 


But by Ci3) I f (y/x) dy X 1 
-oo 


OO o< 

lain* f f Cy/x) dy * p < l , (l6) reduces to 

X 


oo oo ^ 

R.H.S.(14) > P log X X f(x) f (y/x) dx dy 

.00 .oo 


00 00 << oC 

a p log X X f (x) f (y) f (x/y) dx dy 
«oo .oo 


00 


f log ; f (y) 


*.00 


®°oC 1*^ 

X f (x/y) f (x) dx 


.JOO 


dy (16) 


But in 


00 «< 

X f (x) f (x/y) dx 

^00 


-ing k = i/oC < 1 and k** (l/l-°<) so that (l/k>f(lA* )=1 

1 applying Holder inequality for integrals for these values 
the parataeter k, page 140, Hardy, Littlewood and Polya 
52 ) , we iiave 



By this and (14), (16) 





Multiplying (l7) by y a negative no,, and using the 

fact that P 4 : 1 and ^ > 1 , we have 

H^CX/X) ^ B^iY) ^ > 1 (18) 

Combining (l2) and ( 18 ) we have 


(y/X) ^ ^(y) 0 , 1 (19) 


In the limit, as 1, (19) tends to the Shannon* s 
inequality Hj^(y/X) ^ H^(y), so combining this symbolically 
with (19) we have 


\{Y/X> ^ H^(y) 

■<>0 

(20) 

Similarly ^J((3/y) ^ H^(X) 

« > 0 

(21) 


3.2 Belations betMeiBiL..II^(XJa, and H^ (y/X) 


Two infinite sets of inequalities, every member of 
either set tending in the limit, as 1 , to the known 
Shannon* s iequality i|<(X,y) = ^(X) ♦ it((y/X) and two bounds 
for ^g^iXiY)t one upper for 0 < «(< i and one lower for «<J> 1 , 
are developed. 

Case (i) for 0< «« 1 

0000 o( «C 

logf / f (x,y)dx dy » log X / f (x) f (y/x) dx dy 

imSX> ^OQ mX)QmOO 


00 


log X 
.00 


1 

o(- — r 1/k o( 

f(x)*^ f f (x)f Cy/X) dy 

..00 


where 0< k < 1 . 

111., 


dzC33) 
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For 0< k< 1 and chosen subject to (l/k)+(i/lc* )®1 w® 
know by Holder's inequality 


oo 


1®00 


r" 


r_ —1 

f k 

|Vk 

oo 3tf 

X i>i(x) dxj 

«oo ^ J 


X (x)dx 

«00 -s 


l/k* 


5 


«> 1/k e< - 1/k 

Taking ^^Cx) = J t (x) f Cy/x)dy and ® ^ 

X ^00 

in it we have 


°o oc^j/ic’ 

’ oo 3/k 

« 

1 dx;^ 

““('sC-lA)!*:* 

X f(x) 

X f(x) 

f (y/x)| 


X f(x) dx 

-OO 

«oo 

J 


•.oo -J 


l/k* 


(23) 


Using (23) y (22) reduces to 


r k 

* l/k o ( -) 

X X f(x)f (y/x)dy} dx 
«oo i.OO J 


oooo 


<» (‘<-3/k)k» 


log X X f (x,y)dx dy\ k* log X f (x) dx + Vk 




«»oo 


oo 


oo 


log X f(x) I X t (yA) dy/^ dx(24) 

.03 ( .00 


But by Jensen's inequality 

oo C oc ) ^ 

(3/k) log X f(x) < X t (yA)dy) dx 

^ ^<x> J 


k 


oo 


oo 


X f(x) J log X f (y/x)dy) dx 

^OO I .,00 J 


Using this in (24) and then dividing it thro'out by (l/l-«c), 
a positive no., we have 
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H (X,Y) >/ H (X) 4 ^<(Y/X) 

^ (o(-VlE)fc' 

where 0<®c<i, 0<k<l, and lA 4 lA* =» 1 
But for 0< «=< < 1, 0< k < 1 



(25) 


so (“< - lA)k‘ * (l-°<k).Cl-k) lie between 1 and Hence (25) 
gives an infinite no* of inequalities connecting I|jj(X|Y) and 
I^(Y/X) for 0< «(< 1 to H^(X) for «c > 1, It can be easily 
verified that in the limit, as *=< •* 1, all these inequalities 
tend to the Shannon's equality H^(X,Y) = H^(X) 4 H 3 ^(Y/X). 



Supposing k > 1 in (22), choosing k* subject to 
1/k 4 l/k* s 1 and applying Ifolder inequality for this range of 
the parsaaeter k we have 


c< r k» 

^ log / f(x) i dx 

..00 .CO ^ ^oo ^ J 


lA •) * 

+ log / j/ f (y/x) f(x) dyj dx (26) 


«00 mJX> 


Multiplying (26) by (l/i-o() , a positive number, we have 

1 OO ^ 00 k 

H (X) 4 — log X f(x) / X f (y/x)dy] dx (27) 

(«=<-lA)k» (1- «c)k -oo' iloo f 


where 0< «c< i, k j> 1 and. lA 4 lA’^ » i 
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Taking limits of (Sf?) as k we have 


Lt H (X) 


H (X) 
«< 


( 28 ) 


1 <r of ? 

Lt log I f(x) 1 I f (y/x) dyj dx 

k (i»o()k -CO -oo 


e Lt , 
Ic -oo 


OO . OO } f 1 ®°af ? 

I fCx) 1 1 f‘^(y/x)dyj < :r;;2r^S X (7/x)dyj dx 

inaoCID «»O0' MafCXIf 

^ It 

f f(x) 5 X l'^(y/x) dy ? dx 
«oo c*oo J 


(29) 


Since the R. H.S* of (29) is a weighted mean of 

1 <X) ^ 

log X f (y/x) dy » H^(Y/x) , so L.H.S. of (29) 

.00 

< max. H (1/x) . Using this and (28), (27) reduces to 

]^(X,Y) 4 BL(X) + max. I^(Y/x) (30) 

X 

(30) gives an upper bound for I^(X,T) for 0< «c < i. 

,S,as,t II.). to.5. ,!>, ,„1 • 

In deriving (24) we did not make any ccxidition on °t, 
so applying Jensen* s inequality to the 2nd term of its R.H. S. 
and then multiplying it thro’out by (l/l-oC) , a negative no., 
we have 

fl<(x,y) ^ (31) 

where ^ 1. *"^ 1 ^ , 0< t < 1 and lA ♦ lA' » 1 . 



Uxider tbe conditions for whicli (3l) is valid, < 1 , 

so (31) gives an infinite no. of inequalities connecting 
^(X,Y) and 5<(Y/X) for «<> l, ^(X) for 0 < «( < l. 

Every member of this set tends, in the limit as «< •• 1, to the 
Shannon’s eqtmlity H^(X,X) * H^(X) + H^(Y/X). 

fQr ^> . 1 

Again in deriving (26) no condition on ®( was made, so 
miiltiplying it thro ’out by (l/l-«<) , a negative number, we 
have 

k 

^(I,T) > H U) + - — - log t fU) { ; f CyA)dy| ax 
(<<-‘lA)k* (l-®<)^ -o® ^ 

.... (32) 

In the limit, as k , (32) tends to 


7/ I^(X) 4 min. EL(Y/x) (33) 

X 

(33) gives a lower bound for B[^(X,T) for «c > 1. 
•Similarly we csn prove, for 0< =<< i, 


H (X,Y> ^ H (Y) 4 IL(X/Y) 

(«-lA)k^ ^ 


H^(X,Y) ^ H^(X) 4 max. ^(Vy) 

and, for «< > 1, 

^(X,X) H (X) 4 ^(X/X) 

^ (‘<-lA)k‘ ^ 


^(X,X) 4 min. ■ 

y 



i>,^MYJ?,; t„ScLaallty Mt^sen it((X,T) and ^(X)+ H^(Y/X) 
(i) If X and Y are jointly Guasslan, the relation 


X,Y) » q^(X) + H^(VX) s ^^^Y) + i^(2/X:) '<> 0 (34) 

s* The proof of this follows from the corresponding 
It for multi dimensional continuous schoaes dealt with 
action 2 . 4 , part II of this chapter. 

(ii) If f(y/x) is uniformly distributed for each x, 
lall have 


i^(X,Y) = ^(X) + ^^(y/x) 


« > 0 


( 36 ) 


, for «< > 0 , ^ 1 , 


OO 00 


^(X,Y) * ( 2 /l-«<) log X ; f (x,y) dx dy 

*,00 -OO 


*(l/l-®<)log 


00 


OO 


00 


o([ 

/ f (x) dx X f (y/x) dy 

-OO .JOO 


OO 


= (l/l-«c)log X f^ (x)dx + ( 2 /l-<)log X f“^(y/x)dy 

-OO .OO 


1& 


1 ®°o( 1 

log X f (ac)dx 4- X f(x)(logX f (y/x)dy)dx 

1- o( .OO 2.-e< -OO -OO 

H^(X) + H^(Y/X) «< > 0, oc f 1 (36) 


) * Hj^(X)+ H^(y/X) is known to be true, so 
ibolically with ( 36 ), we shall have ( 36 ), 
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Similarly if f(x/y) is imiformly distributed for 
each y, ve shall have 


H^a,T) = H^CT) H^(2/Y) <<>0 (37) 

(iii) In general 

H^(X,Y) # fi^<X) * ^(y/X) k ‘=<> 0,o( ^=1(38) 


Consider 


-x(y-n) 

f(x,y) = X e 

« 0 elsewhere 
The marginal densities are 


0, y^ 0 


f(x) s e 
= 0 


f(y) 


(1+ y) 


2 


0 

elsewhere 

y>o 

el sewlwre 


and the conditional densities are 


fCy/x) * X e*^ for all x^ 0 

f(x/y) » (1 + y)^ X for all y^ 0 


So 
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1 <»co « -X <<(7+1) 

H^(X,y) = log X ; X e Cbc dy 

^ 1- 0( o O 


0(4,2 

5 J — — iQg i 4 . log /7(a( + 1 ) ocJ> o,o( 4 1 (39) 

l.c( ^ 1- 0( 


00 -0(3J; 

I^^CX) a (Vl-^) log X © <351: 


a (1/1-«C) log 1/«C , «> 0, «C ^ 1 


(40) 


°° dy 

H_(Y) a (l/l-«^) log X , §5“ 


o (l+y)*" 


(41) 


Ihe integrand in (4i) is convergent if 2=<J>li*e, 
°( > 1/3 ; so for << ? 1/2, °c # 1. 


H^(X) * (Vl*-<) log (1/2=C-1) , oc > V2 , “C ^ 1 (42) 


^(Y/X) 


« (l/l-^C) X © 

o 



(43) 


1- « 


H(X/3f) = 


1 ^ 

1-oc o (1+y) 


2 


-«(x(y+l) 2=< ”] 

log X X e (1+y) dx| dy 


I 


r 


- 00 1 I -o(-l o(»l 

X 2 log r(«+l) «c (1+y) 

1- «< o (1+y) 


J 


dy 
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1 n 

* log + log =< - 1 , «c > o, «c 4: 1 

1* ®( «3(-l 

... (44) 

From (39), (40) and (43) we have In general for > 0, 

■=< t 1 


i^^(X,Y) # H^(X) + H^(Y/X) (46) 

and from (39), (42) and (44) we have In general for ^ > 1/2, 

°< 4 * 1 

^<(X,Y) # ^(Y) + i^<(X/y) (46) 

and from (40), (42), (43) and (44) we have in general for 

< y 1 / 2 , ^ 1 

Bt((X) + ^(Y/X) t H^CY) + ^(2/Y) 

(38) follows frcm this and (46) and (46). 

4, aOME MATMATICiL SIFHCDLYIES IM ASOPYING H.(X) 

m ,A,.M J 4igias QJ 

aenyl*s ‘^-entropy given hy (i) for <ilscrete seheme 
is positive, finite and invariant under the transformations 
of coordinate systems* It will be shown tlmt, in ccsaparison 
to it, its extension ^(X) given by (4) for a continuous 
sch^e has noi]^ of these basic properties. 
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4.1 may be negative 


Considering f(x) 


3 


0^x4: 


S 0 


elsewhere 


H^(X) * 


1- «( 


log ; ( 3i®/a®) dx 


1 S* 

log ( ) 

1 - «< 2 “C+l 


So from (47), for 0< «( < l , 


H^(X) > 0 

if 

a > ( ■ 

m 0 

if 

a * ( ’ 

< 0 

if 

a < ( 

and) for ^ 1> 



I^(X)7 0 

if 

a < ( 

» 0 

if 

a * ( 

< 0 

if 

a ^ ( 


2 ‘<+1 ^ l/(l-°{) 

at ^ 


2=C + 1 l/(l-a() 

2°< ♦ 1 1/(1-«C) 

_ — y 


2 «( ♦ 1 i/(i-'=<) 
) 


ar< 


(47) 
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^(X) may be infinitely large 

Let X, having the prohahility density fCx), taike^ 
a continue of values in [a, h] so that 

/ ® 

P(1 < X <ift ) S ; f(x) dx = F(m) - PCD 

1 

Divide (a>b) into n+l non-overlapping suhLntervals (a, a^^ » 
» ••••! (a^» h^ and let a^^ " a = ^a^^, 

^ • «n * ^ Vr ^ 

®1 

P(a < 2 ^ a^) « X f(x) dx « P(a,) - F(a) * p, ^ a, 

a •‘•1 


P(aj^ <X 4: b) « 


b 

I f(x) dx * 


P(b) - 


F(a„) = 
n 


p ^ a 
n+l n+l 


Define another randosi variable (say) assuming only 

the discrete set of values |a^, a^, with respective 

probabilities 




} 


%+i "^®n+i 


(48) 


n+l 

such that S p, Aa, a P(b) - P(a)=l- Dy (48) , 
1*1 ^ ^ 


Renyi’s '^-entropy for the variable X^i is given by 

«( o( 

Cl/l-=<) log Z p^ (^9i) 

1*1 i ^ ^ 


( 49 ) 
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tfiader the conditions it is reasonable to expect (49) 
approach, m the limit as n ■* «>, to the “(-entropy associa- 
ted with the continuous variable X. Letting ^a^ * 

“ I it reduces to 


/ ^ . “+1 oc 

(l/l-«) log ( 2 p^ 

1»1 i 


+ (l/l-«() log ( i^x) 


n+j. 

a - log Ax + (!/!-*<) log ( s p. ) Ax 

i*l i 


(60) 


In the limit, as n *♦ ®®i*e*, Ax *-0, (50) reduces 


to 


L c( 

- Lt log Ax (l/l-C) log X f (x) dx (51) 

^x - 0 a 


So in the limit, idaen an infinite no. of infinitesimal 
subintervals are a>nsidered , H^(X) given by (5i) becomes 
infinitely large for 0. 

4.3 H (X) may not be invarian t under a transformation of 
coordinate systems. 

Let the variable X be transformed into a new variable 
Y by a continuous, mono tonic snd 1-1 transformation Y * g(X). 
Ihe density fuiiction ?(y) of the variable Y in terms of 
f(x) is given by 


P (y) * f( 2 :) |dx/dy| 
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So the «c- entropy associated with Y is 

of *<-1 

iL(Y) « (3/l-««) log X f' (x) |dx/dyl dx (62) 

^ -oo 

The Value of the integral on the R.H*S. of (62) depends 
upon |dx/dyj or |dy/dx| ^ a function of x. 

Wot exaiBiple) letting Y * AX Bj we have 

\iT) s H^(X) + log |Al 

5. DEJIIIITIOHS AHD PRDPERTlFS OF 

As shown in section 4 E^(X) does not possess the 

three basic properties of being positive, finite and 
invariant under even linear transformations, this 
obscures its significance. But , nevertheless, it retains 
its importance , for the =<-infoi®ations defined below depend 
on the difference of two “{-entropies. These “{-informations 
possess the above basic properties and may be termed as 
informations of order “£ processed in the channel. 

^ may define “{-information, ®{ 0, «{ 5^ 1 , conveyed 
about X by Y as 

I^(X JY) * H^(X) - H^(X/Y) (63) 

and "{• information, ®{ > 0, ®( ^ 1, conveyed about Y by X as 


I^(Y| X) « ^<(X) - ^(Y/X) 

Note that in general lojCX? T) 4 le((Xj X) 

^ ( 38 ;! 


(S4) 
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We may term I^CXjY) and Ia((y;X) as ^^-informations 
of type I and type II re^ectively. 

In the liffli.t, as “C •• 1 , both I^CXj Y) and I^^CY; X) 
tend to I;j^(X » Y) ^ trar^nformation for continuous channels. 

S*1 I,jj(X;Y) and Ioc(Yj X) are always positive 

This follows from (20) and (21). 

Hote that Ig((X | Y) and Igj( Yj X) are zero only when 
X and Y are independent. 

6,2 Io((X } Y) and Io((Y ; X) are generally finite 

When from (60) the expression for i^(X) was approxi- 
mated as a limiting case of the corresponding expression for 
the discrete case there appeared in ( 6 l) a term 

- Lt log AX ( 66 ) 

Ax -*0 

that led us to say that H^(X) can he Infinitely large. So 
when an expression for Io((X ; Y) or Io<(Y j X) Is approximated 
in the same way as a limiting case of the corresponding 
expression for the discrete case there would appear two 
same terms of the type given in ( 66 ), one with the positive 
si 0 a and the other with the negative sign j they would cancel 
each other and thus the resulting expression for j Y) or 

Ia(( Y 5 X) would generally he finite. 



Ie((X ; Y) and ; X) are invariant iinder linear 

ja?aiaaformatt9n«; , 

In the original coordinate system let X and Y have 
the densities f(x), fCy), f(x,y), f(y/x) and f(x/y) 
subjected to 

X’*aX+b , Y*»cY + d (56) 

let the above densities be respectively transfowaed to 
$(xM» (l)(y‘), f)(x*,y*)i ()(yVx’)» (>(x*/y')» thai we know 
that 


a(x»,y») 

f(x,y) * p(x',y»)| 1 « ^(x',y' )|a c| , 

d(x,y) 


fCx) * p(x')|a| etc. 


So 

1 of 1 °° 

TLil/Y) * log / f (x)dx / f(y)(logi*f^Cx/y)dx)dy 

^ 3^-oc -oo X~ 


1 a( “<-1 

*s l og / p (x*)ja{ dx* 

X» o< -oo 


1 °° 

— ; f)(y’ ) 

1- 0< -OO 

^ cl =<-l 

(log X <|> (xVy*)|a| dx‘)dy* 

.OO 


1 ^ <v 1 *^“1 

log I (>''(x»)dx‘ + logjaj 

X-0( -OO X -qC 


X ««-l 1 °° of 

log|a| X f>(y')ClogX ^ (xVy’)dx 

X-o< It *< -®° 

dy^ 




\{X*) - (X* A' ) 


(S7) 
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Similarly we can prove that, subject to linear 
transfoimation defined in (56) 


^(Y) - V^{J/x) = ^^(y» ) - ^^(Y'/X‘ ) 


( 58 ) 


6. 

Iwo types of =<-infoimations defined in section 6 
in a natural way give rise to two types of =<- capacities, 

(l) (2) 

and , So *<- capacity of type I may be defined 

as the maximum of «c- information of type I and “(-capacity of 
type II as the maxlmm of “C-infoimation of -type II over the 
whole class of admissible input probability density functionst 
Symbolically 


C » max 
f(x) 


. I^(Xj Y) « mape. |\U) - H^(a/I^ 


(59) 


and 


a m^. I^(Y5 X) * ffiaJC. (Y) - ^(Y/X)J (60) 

(l) (2) 

In the limit, as «( •♦ 1, both and tend to C^. 
Since in general I^CXj Y) J so general 


Xl) 


♦ 


( 2 ) 


7 . tLTBBllAnv ii lEmalrlOH OF OOIiBrilOliA «-MlBDF? MD 

vEiagCAiioN OF Bsajua 

A3 in the discrete case, conditional “‘-entropy t(T/X) 
can be defined in another possible way. Instead of taking the 



mean value of the way we did for deriving (9) 

from (8) we can also consider again the strictly mono tonic 
and continuous function g^^Cx) given by (5) for ’ 'information 
contained'* variable I^CT/x) and take the generalised mean 
value of ^(I/x) w.r. t* g^ix) as we did implicitly for 
getting (8) or explicitly for getting the extension (6) of 
Renyt's ‘^-entropy. So if we do this, the conditional 
entropy would be 


H^(YA) » g; 


-1 


oo 


(l-«C)^CY/x) 

X f(x) (2 ^ ) dxl 


.oo 


<6l) 


Using (8), (6i) reduces to 


oo 


H (Y/X) » gQ 


•1 


logo X_i^(y/x)dy 
X f(x) ( 2 ^ ) dx 


..OO 


OO 


oo 


'(l/l-®t)log_ X fCx) ( X f^Cy/x)dy )dx 

> 0, oc ^ 1 


(62) 


Based on the same procedure as adopted for deriving (62) 
we have 


&& oo 

H (X/Y) ®(l/l-<)log^X f(y)( X f‘^(x/y) dx ) dy (63) 

o( ^ 0, ^ 1 . 

In the limit, as «< •♦ 1, (62), and (63) tend to the 
corresponding Shannon's definitions* 
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But by Jensen* s inequality 

oo oo oo oo 

X f(x) Clog^ X f‘^(y/x)dy)dx < log- X fCx) ( Xf (y/x)dy) dx 
-oo 2.00 ^ ^-oo -OO 

so for 0 < «< < 1 , 

OO oo °° 

rniXlog^ ; f°‘(y/3c)dy)to ^ (l-=c)logg SJM 

cx> 

( X f‘^<y/x)dy)dx 

*oo 

(64) 

and for *< )> It 

oo oo 

(l/l-<) X f(x)(loggX^f‘'(y/x)dy)dx > (l/l-=<)log2 

oC 

( X f (y/x)dy)dx 

•OO 

* • « • (65) 

By (64) and (65) conditional =<-entropy given by (9) 
is less for 0< «< < 1 and more for It than that 

givai by (68). Similar interpretation can be had for (lO) 
by comparing it with (63) by Jensen* s inequality. Based on 
this it follows that «<-infoimations defined by (63) and (54) 
would be less for 0 < « < 1 and more for <<>1 if we measure 

H^(Y/X) and ^ ^ 

Hesults, which involve specifically H^(T/X) or ^(2/Y) 
and given by the following equationsi are also true if we use 
the possible alternative definitions of H^(Y/X) and l^(X^) 



- 129 - 


given respectively by (62) and (63) s 

(aO), ( 21 ), (30), (33), (34), (35), (37), (38), (43), 

(57), (S8). 

In fact ail the results that involve either H^(Y/X) 
j^(X/X) and are developed in earlier sections, except 
those given by (25) and (31), are true for these alternative 
definitions,. 

|?Qte the generalisation of results developed in 
section 3 for suiy finite nmber of lanivariate continuous 
sch^es are not talsen in this part since these general!* 
sations also follow from the generalisations for any finite 
ntsuber of multivariate continuous schemes of tl^ corresponding 
results for two multivariate continuous schemes. These 
corresponding results and their generalisations are taifcen up 
in part II. 

It may also be noted that the results developed in 
sections 2.4, 2.5 and 3.4 of chapter 2 can also be extended 
to the continuous case. 


1, ththodpgtiqn o(«ENTaQPY kBsooikmi..Min A Raim miaa. 

Suppose the input of a channel is an n-dimensional 
continuous random vector X * having the proba* 

bility density f(X) positive everywhere in its domain of 
definition ]^ - an n-dimensional rectan^e. Following the 
procedure adopted for getting (6) of part I if we consider the 
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strictly monotonic and continuous parametric vector function 

( 1**^) X 

g^(x) =2 Of < ^ 1 (1) 

and take the Rolmogorov-Sagumo generalised mean value of 
logg(l/f(x)) w.r.t. (i) we shall obtain =c-entropy associated 
with X • Hence 


H (X) « 
0 ( 



(l3^)l0g (l/f<x)) 
X f(f) 2 ^ 

% 


= <l/l-«() logg I f (k) dX «<> 0, « # 1 (2) 

®n 

Mote that either by adopting Shannon’ s convention 
that 0 X 0 or by considering the evaluation of 
in the extended real number syston the definition (2) can 
be made valid even when f(x) vanishes at the most over a 
set of measure zero. 

In the limit, as 1, (2) tends to the corresponding 
Shannon’s definition. 

% taking (2) as the basic measure of entropy some 
basic concepts of multidimensional continuous memoryless 
channels have been generalised. All the results developed 
below teni in the limit, as ■=< *• 1, to the available results, 
fhe results needed are t 



(i) “Jensen's inequality for multiple integrals'*, 
page 21| Morrey (1966), Since the interpretation of the 
result given there demands that the means of tdae components 
of the vector, of whose convex fmction the theorem deals 
with, should exist, so our results would he confined only to 
such distributions. 

(ii) '' Bolder inequality for multiple integrals'*, 
page 21, Beckenbach and Bellman (1965). There will be strict 
equality in the results of B)lder inequality when the under- 
lying multidimensional schemes are independent. 

It may be noted that the brief proofs of the results 
in sections 2,2 and 2.3 are given in spite of the fact that 
these results are straight forward generalisations of the 
corresponding results dealt with in part I because of the 
following reasonss 

(i) the proof for the result in section 2.2 is 
applicable whether the mderlying channel is with memory or 
without memory whereas the proof given for (20), part I, is 
applicable to channels without memory* 

(ii) the proofs of the results given by (60), (6i), (62) 
(63)and (€^) are based respectively on the lines of the proofs 
of (24), (30), (31), (33) and (60); if the complete proofs of 
the former results are resorted to, the presmtation would be 
much lengthier that what it is; so to save space, to make the 
presentation easily assimilable and more or less independent of 
its two dimensional counteipart dealt with in part I, the 
brief but complete proofs of the latter results are given 
whereas the proofs of the former results are simply outlined. 
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It may also be noted that the operations dealt with 
below are contingent upon the existence of the mderlying 
integrals* Prom section 2.2 onwards* base of the logs 
will be assumed to be e. 

In 2,1 , oc-entropies associated with a multidimensional 
continuous chaniiel are defined and in 2,2, 2,3 and 2,4 seme 
relations between than are developed. 

2*1 

Let the input of the channel be an n-dimensional random 
vector X » with the density function f(x) 

defined over an n-diiaenstonal rectangle and the output 

be an m-dimenslonal randem vector Y * j^Yj,,,.., with the 

density function f(S^) defined over an m-dimensional rectangle 
% . Ta^en jointly both input and output wotild be defined over 
the subspaces of Idle product space fhere are five «C- 

entropies associated with the product space representing 

the channel, 

(i) ^(i), defined for the input X. It is given by (2). 

(ii) H (Y) , defined for the output Y. Based on the 
procedure adopted for getting (3) it is given by 

H (Y) » (l/l-^) lofe ! f (y) ^ 0, ^ 1 (3) 

(iii) H^(X, Y), defined for the system as a whole. 



Based on the procedure adopted for getting (2) it is given by 


Y) s lofo I f (x, y) dx dy (4) 

(iv) B^{ Y/j), defined about the output Y when 
it is knoiiQ that X is transmitted and may be termed as 
conditional ‘’^-entropy# It may be defined as such s 

Based on the procedure adopted for getting (2)| 
conditional ^^-©atropy calculated on the assunptlon that the 
event x of X has occurred is 

H^(T/X) * (l/l-°c)log 2 X f^(y/x) dy , > 0, # 1 (6) 

\ 

To have a measure for Y/Z) it is quite natural to take 
the expected value of (6)? so 


TS^iX/ 1 ) » Cl/l-^) / fCx) 

% 


X if/t) dy 

L ^ \ 


dk 


( 6 ) 


« > 0, "C 4 1 . 


(v) t defined about the input t when it 

is known that Y is received and be termed as «(-equi vocation. 
Based on the procedure adopted for getting <6) it is given 
by 


» (l/l-<) I 


log« X f^Xx/y) ax 


df 


(7) 


c<> 0, OC # 1. ; 

It can be easily verified that in the limit, as •* 1, (3), (4j, (6 
and (7) tend to th® corresponding Shannon* s definitions. 



since 


so 


Lt 

o( -• OO 


J* f(x) \x) <si 




% 

L 


as max f(x) 


Lt H (X) * • log^ max f(x) 

«»# tso ^ G 


«« ® *^062 

"Ot ’**1^ Ww ^ 

x^y 


Lt H.(I/X) as - / f(x) 

a( ^ OQ *< 




log^ max f(y/x) 

, ^ y 




2.2 aelatlon JBetween ^^(Y/1) And qbf(Y) 

Case when 0 < *< < 1 

0( ^ ^ 

Since log X t (y/x) dy Is a concave function 
of X, so "by Jensen’s Inequality for multiple Integrals. 


I m) 


.log f (y/x) dy 


dx ^ log X X f(x) f (f/x) dx dy 


log X 


1“®< 0( 

X f (x) f (x,y) (&| dy 
»n 

... (8) 

l-*< at 

In X f (x) f (i»y) dx, let i/p * «( and l/q « l-«< 

B 

n 

so that l/p + l/q » 1 and applying Holder* s inequality for 
multiple integrals for p ^ li page 21, Beckenhacb and 



BeXlniap. (1966), we have 


1-Q( o( 1*^ *< 

S f (x) f (x,y) dx X ( X f(x) dx) (X d*) 

a (y) (9) 

Substituting (9) in (8) and dividing the result by (l/l-«c), 
a positive ntanber, we have 


H^(Y/^) ::C 0 < «C < 1 (10) 

Case when «( > i 

Since 

So 


0([ 

X t (y/x) dy ^ ( X f(y/x) df ) = 1 


X f(x)(iogx /‘(f/xHfmy, X m) ( x f^(y/x)dy;iog 

IL. K- R S 


% 






(X f (y/x)df) 


dX 
..(11) 


Since ( X /^(f/x)dy) log ( X f^(^/x)dy) is a convex 

function of x, so using Jensen's inequality for multiple 
integrals (ll) reduces to 


X f(x) ilog X f (y/x)dy)dx> 

HR 
n m 


X f(S;)( X f (y/X)dy)dx 


S m) a 1 (J/x)dy )az 


log 


( 12 ) 



But X !*(?/*) dy^ 1 i let X f" (y/i) if = P- (say)^ 1. 

\ \ 

Let P *e min. P- 5 using this (12) reduces to 

X •*■ 


X f(x)(log X t it/±m )dx> p log X ( J f (x)f(X,^)dS)dy 

®a ^ % % 

...(13) 

1-°C 

In X f (x) f (3c, y) die let 1/p » *< and 1/q = l-«( 

Sn 


so that l/p + l/q =» 1 and applying Bfolder’ s inequality 
for multiple integrals for p < 1 , we have 


X 

R 


l*®t a( 

f (x)f (x,y)dx> 


n 


X f(x) 


l-oC 

dx) ( X 

Sn 


m^f) 


oC 

dx ) as 


o( 

f if) 
(14) 


Using (14) in (13) and multiplying the result so obtained 
by (V'l-®^) » a negative number, we shall have 

H^( Y/X) ^ p ^(Y) (15) 

Since p ^ 1> so frtma (l6) we have 

S *<> 1 , ( 16 ) 

Combining (lO) and (I 6 ) we have 

^(Y/X) ^ H^(5) « > 0, # 1 , (17) 


In the limit, as « 1, (l?) tends to H^(Y/X) ^ H^(f) 

as already kno«i as fijiannon’s inequality, so combining this 



syaabolically yith. (i7) we shall have 

^ (i) °c > 0 (18) 

Similarly we can prove 

\im) ^ ^^a) <<> 0 (19) 


2.3 Belations Between H^(i,Y) , H^(i) Ahd ^(Y/X) 

Two infinite sets of inequalities, each member of 
either tending in th© limit, as “< -♦ 1, to the known 
equality H^(X,Y) * Ej^(X) + H^(t/X), and two bomds for 
I^(X,Y) , one i^per for 0 < «< < i and one lower for 
y It are developed. 

Case (i) for 0 < «< < 1 


log ; f^(x,t)di if 


log ; 


c<-l/p 

f(x) 


r vp 

im) 


n 


0( 

f (Vx)dy 


di (20) 


where 0< p < l. Choosing q subject to l/p + l/q * l 
and applying Holder inequality for multiple integrals for 
this value of the parameter p we have 


log / f (x,f)dk dy> log 


14-n 


«^l/p q 

! (f (i) ) dx 

H 

n 




+ log 


Vp o( P 

X ( I f (x) f (y/x)dy) dx 

^n % 


o e e C^l) 
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But by Jensen’s Inequality for multiple integrals 


log X f(x)(X f (y/x)d:^ ) di p J m)(log X Ay/x)dy)dx 

^ ^ Bn «m 


.... ( 22 ) 


Using (^}) (2i) reduces to 


f -v 1 (‘^“’l/p)q of 

log X f (x,y)d 2 dy > i log X f (x)dac + Xf(x)(logXf (y/x)dy)dx 
^+n Bn % % 


(23) 


Multiplying (23) by (l/i-o(), a negative number > we have 


. ^(f/S) (34) 

for 0< «( <1, 0 < p< 1 and l/p + i/q s i 

Under the conditions for which (24) is valid, («<- l/p)q 
lies between 1 and o® . So (24) gives an infinite set of 
inequalities connecting ^(X, y) and H^(Y/S) for 0< «« 1 
to ^(X) for «C> 1. It can be easily verified that In the 
limit, as “C 1, every member of (24) tends to the known 
equality H^(X,Y) * H^(x) + H^(Y/X) . 

Case (ii) for 0< *<< 1 

Choosing in (20), p^ 1 and q subject to 1/p + l/q » i 
and applying HSlder inequality for multiple integrals for 
this valtie of the parameter p we have 



3/p 

r i/p c< p 

+ log jx ( X f (i)f (y/X)<i^) dxj ( 26 ) 


Multiplying (26) toy (l/l-«<), a positive number, we have 

H (i,t) H CS) 4 -J— log I f(x) [; f‘(y/S)dyl d£ (26) 
(a(-3/p)q (l-oC)P I B J 

L ® 


for 0< «C < 1, p ;> 1 aud l/p + l/q =1 
Talcing limits of (26) as p we have 


Lt H (i) a H CX) 

P ^ 00 (o(-l/p)q 


(27) 


Lt. log X f(x) 

p oo (i^)p 


X f (y/x) dy I dx 


* Lt X 


f(x) ( ^f (y/x)dy ) 1 


(y/x)dy) 


P X f(x) ( X l®^(y/X)dy)P dx \ 




The I.H.S. of (28) is the weighted mean of 

1 log X f^ (y/x) dy i.e. H (Y/x) , so 


... (28) 


L.H.S. of (28) ^ majc ^^(f/t) 

X 


( 29 ) 
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Using (27) and (29), (28) in the limit, as p reduces 

to 


f) H (X) + max H ( T/x) (30) 

X 

(30) gives an upper bound for H (X, Y) for 0 < «< < 1. 
Case (i ) for < 1 

Since in deriving (23) no condition on was made, so 
multiplying (23) by (l/l-«(), a negative number, we have 

H (i,Y) ,< H (X) + H.(Y/X) (31) 

for *<>11 0 < p < 1 and l/p + l/q * 1 

Under the conditions for which (31) is valid ('<-V'p)q < 1; 
so (3l) gives an infinite set of inequalities connecting 
%(X,Y) and H^(t/X) for 1, to 1^(X) f or 0 < << < 1 . 

It can be easily verified that in the limit, as *< -• 1 , 
every member of (3l) tends to H^(X, Y) « H^(X) + H^(Y/X). 

Case (ii) for 1 

Multiplying (26) by (l/l-«c), a negative number, we 
shall have 

H (X,Y) > H (I) + log X f(x)( f ftvx)dy)^dx 

(‘<-3/P)q (1-°«)P \ \ 


... ( 32 ) 



In Hie limit, as p -» «>, ( 32 ) tends to 


H^(X,f) i) + min. H (Vx) (33) 

X ^ 

(33) gives a lower bound for H (X,Y) for «< y 1* 

o( 

Similarly we can prove that for 0 < =< < 1 , 

H^(X,f ) ^ 0< p < 1 , 1/p + 1/q » 1 

I^CX,Y) ^ ^(f) ♦ max H (^y) 

y 

and for / 1 , 

H (x,y) 4 : H (y)^ 0 < p < 1 , l/p + i/q * 1 

(o(-l/p)q 

H (X,Y) ^ ^<(X) + min, is/f) 

7 

2.4 A ,a£Hdy,„pOa3ia;^.te 

It will be proved that if the joint distribution of X 
and Y is gaussiaP then, for <> 0 ^ 

H^(X,Y) a ^(i) + S<(Y/f) a H^(Y) + ^(S^) 

An example that this is nothin general^ true is also indicated 
^n the end. 

Below, throughout this part, a single integral siga will 
stand for a multiple integral and the number of variables for 
which it stands is understood from the dimension of the varialale 



vector. 


Let 2 be the random vector describing the joint 
behaviour of X and t i.e. Z * Tx, xj > 

Y^,. ij • ^thout loss of generality we may assume 

that fi(i) = 0 and SCY) * 0 . 

With this the probability density of Z is given by 


f(i) 


1 


Cm4'n)/2 1/2 

(2v) ICI 



1 ( 2 * 
2 



)\ 

J 


where z^is the transpose of 

by 

'A 

C s 

D* 


z and the matrix C is given 



In the partitioned structure of C, the matrix D’ is the 
transpose of D and A, B and D are respectively given by 


A = [hjl • “U = Jo ^ *1 


B = ’ h.i “ Lo 

m 

» » [‘ijl > "13 = J= *1 =^3 

the marginal densities for X 


fCx) dx i,3 = 1,2,. ..jn 

f(y) =4 y i»5 * l, 2 ,...,m 

f(z) dz i- li***jnj 

js* 1, ...,m 

and Y are respectively 



given by 


f(x) 


n/2 1/2 

Or > U| 


exp, 


1 . -1 1 
— (x ‘ A X ) j 
2 J 


and 


fCy) 


m/2 1/2 

(2r) jB I 


exp, 


1 ^ 'i 

( y' B y) i 

l 2 J 


The conditional density of Y given a value x of X Is 
given by 


1/2 

f(x,y) f(z) |a| 

'“‘■ IS - ■ 


-1 

Let C » 



^12 

[aai 

%2 


J 


r 1 -1 -1 . 

exp j-.~(z*c 2 -x’A x) 1(34) 

4 S» 


SO that 


J 


C C » 


A fi 

D* B 


^11 ^12 
^21 ^22 


“I 


'n ''nxm 


®mxn 1 

J 



ASi^+ DSsi = In 

(36) 

BSgg - 

(36) 

®’ *12 ♦ ® ®2S = 1=. 

(37) 

D' + B » 0^ 

(38) 

Solving (35) and (36) we have 


^ ^ “ ^11 * ^12 ®22 %i 

This would make 

v.,«-i. V., -1^ r^, -nf^ii ^2] 

Z»C -^z * x’A X a x‘ yM ^ j 

^ -* S2I SgsMyJ 

1 * ^%1~S12^22 


» (y + $22 ^2^^ )’ S22 ^y ”*^^22 S21 x) (39) 
Hguco ‘^■=^ ^ f 


1 

^ CfiY ) » log 


00 1 

r ( — - gir r ' Ti/ a 


^ mm ^ Qj[ 

exp (- 1/2 [x y] C [x y])) dxdy 


(40) 


Putting in (40) X » P and T * Q > it will b© simplified 
to 


H ^( i , I ) 


m^-n 

2 


log (2r) jCl 


1/2 


m+n 


2 ( 1 -^) 


log *< (41) 



Putting in (42) vT^c x » p , it will be simplified to 


( 42 ) 


_ n/2 V8 n 

^(X) « log (2r) jA| log o( (43) 

2 ( 1 - o () 


Similarly 


m/2 1/2 m 

MY) * log (2ir) 1B| - log o< (44) 

2(1 -=<) 


Using (34) and (39) we have 


oo 


^( Y / X ) 


• f f(x) 

1 -<s( -oo 


oo 


log I ( 


1/2 

Ul 


-«»(2ir) ICf 


1/2 


1 -1 

«xp. - gCy+SgaSgj^x)' 


(T + sj ; So , x ) lax 


’22 


’22 21 


J 


1 - « 


log 


Ul 


1/2 


n CD 

X ♦ . I f ( 3 C ) 


“ Z. 1 - -OO 

( 8 r )'^ IC | ® 


w 0( ^ 

(log f^e3cp.( - (y+S22 Sg^x) S22ar4.S22S2iX))(^)dx 

... (45) 

putting /**< (y + Sgg X ) » f in the second term of (46), 
it ^411 reduce to 



14b - 


1/2 

°c I a 1 1 !^ 

H (t/X) = log — — log «< 

l-^t (2tr^2 |c{^2 


1 m/2 l/2 

log (2ir) 1^22^ 

1- o( 


( 46 ) 


“1 • -1 

By solving (36) and (3?) we have * B - D A D. 


Letting L s 


-(a“^)’ d 


, ¥e have 


L G L = 


I 0 
-DU'^ I 


A 

D’ 


D 

B 


I -(a“^)* D 


U 


4 


0 


-D» a“^ 


+B 


Since A ^ is symmetric 


-1 


so 1 Cl * 1 a| I B - D’ ^ D I 


-v. 


Hence (s^^l 


-1 


1/2 


= [B - D*A D{ * 


Id 


3/2 


lAl 


V2 


Using (47), (46) reduces to 

m/2 3/2 

(2v) I Cl 

H^(VX) * log — ITi 

I 


HI 


2(1- «C) 


log “C 


(47) 


(48) 





similarly m can prove 

n/2 1/2 


s log 


(2v) 


ic| 


n 


iBr- 2(i-<) 

Bslng (4i), (43) and (48) ve have^ 


log «=< 


(49) 


a H^(l) + H^<f/x) «c> 0, o( 1 (60) 

and Using (4i), (44) and (49) we have 

\U,f) a ^(?) + H^(2/x) c<> 0 , oc 4 1 ( 61 ) 

(60) and (6i) gives for 0, ^ 1 , 

H^(X,1^) = ^(i) + ^(y/x) a ^(2) ^ 

From this and the fact, that this is true for “C ® 1, the 
required result follows. 

But in general, for °c > 0, «< 4 i, 

i) 4 + :^(t/x) 


It Can he shown by considering a simple example. An example 
in this regard is the bivariate scheme given by 


f(x,y) = 2 


for 


{ 


0 < X < 1 

0 < y < X 


s o outside 


3. 

In this section various results developed in section 
2 are generalised for any finite number of multivariate 
continuous schemes. 

Consider r spaces of dimensions 


**2****** ^'r 


wildi respective continuous variable points 
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^1 = 


X^l , X^2 




, Xr2 ,Xr n^J 

having the respective density functions f(x^), •« . ,f (x^^) 
so that 


f f(x ) dx- s 1 , S f(x^) d&- ^ 1 

\ \ 


The product space of these r spaces is of dimension 
n^ + ng + ..... + Uj, and it may be symbolically denoted 
\ +02 + +11 * variable point 


^IV ^In^ * ^i» • 

Xra» 

for the product space have f(z) as its density function 
so that jf f(z) d z * 1 . In this product space the 

following are a few generalisations; along with either 
complete proofs or sufficient indications in that direction^ 
of the various results developed in sections 2.2, 2.3 and 
2 . 4 . 
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(i) The inequality in (18) willj generalised to 
inequalities of the foim 


1^ Xgj • • • (62) 

<y 0 

Proof of (62) \ 

l»«p> 

On the assiiaption that an event 

of the product space IL has occurred, let 

the density function of the events belonging to % +n- ^6 

1 * 

given by 

fCiCp Xp /XgtXg,...., Xy.^) * g(X3^, Xy) 


and the schemes given by the variable points and ^ 
become the schemes given by the variahleA X.j =rx*^ 2 .» •»^lnTl 
and Xj, * ... j with respective density functions 


and 


f(X^/Xg, Xj..^) « g(X3^) 

f(Xy /X2f*»^-x) = sCXy) 

By (18) for ^ 0^ 1 we have 


^<(XJ /Xp ) 


But 


(53) 


H (Sj, /Sl)^ — t g(£i) (log S g^Ci/Si) (54) 


and 





giX^x ) = — ss « « ♦ « *^r-i^ 

g(X^) fCX^/XgjXg,.., , Xy.3^) 


~ f CXy/x^^j * . > 

Using (56), (64) reduces to 


( 56 ) 


H3<(^X^) = r 

®n, %« 


Also 


... (56) 


i^(^ ) X log X g‘^(x ) dXp 


1- < H, 


n. 


1 o( 

log I f (^/i 2 ,...,Xj.. 3 ^) dXj 


!-«< 


(S7) 


Using (66) and (57)) (63) reduces to 


1 ^ 

X f(Xn/xo,..,x )(log X f (xVx^, 

1- OC 1 2 \ 




1 « 

log X f (Xj/x 2 ,,...,Xp.^)dXy 


(58) 


Multiplying (58) by ^(Xg, integrating over 

Ho{(Xy/Xj|^)X2» •• • »3C];.»l) ^ B!B((Xy/^» .. . »»Xj,^^) (59) 

=< > 0, «C ^ 1 , 



In the limit, as «t ■* i, (59) tends to the known result 


• • • I %(Xj/X 2 , .... . Ctomhining 

this symbolically with (59) we shall have (52). 

(ii) For 0< =< < i, the inequalities in (24) will 
get generalised to inequalities of the form 

• • • • I H ^ , + H (Ife /%.- ) + .... 

(-«-Vpi)pi C^c-VPgTl’,' ^ 




+ »^-l^ 


( 60 ) 


where 0 <p^,..., < 0 and l/p^ + l/p^ = 1, 

1 1 

+ =* 1 and the inequality in (30) to inequalities 

Pr-1 ^r-i 
of the form 

H^(X 3 ^,...,Xy) 4 ^<^X 3 ^)+ max + max 

\ X^^Xq 


,* max 


.>x^- 


i^(^ji/x2^, > « . • ,Xp„^) 


(61) 


r-i 


(iii) For 1, the inequalities in (3l) will get 
generalised to inequalities of the form 







!»•••» 


^ H 

(c<-l/p^)p 


(X,) + H 

{ ^ ('^-l/Pg)? 


, (V^i. )■<■•• • 
2 


+ ^(Xp/X^j C®2) 

, 11 11 
where 0< . . ,Pj..]^< l and - , + * i 

?! Py-l Pr-i 

and the inequality in (33) to inequalities of the form 


1^(5;^, ..,Xp) ^ i^(X-) + min I^(X 2 /£)+_iBin H 

Xi Xl >*2 


+ . . . .+^ min _ go(<Xy/jt, ,X 2 , . . »x t ) (63) 

A «>L» 

(iv) when the vector variables X^^, . . . are 
normally distrilxited, the equality in (60) will get 
generalise! to equalities of the form 


^(Xi, . . ,Xr)=^<(X^)+ :^(^X^)+. . ,+I^<(^X^, . . . ,Xr.i) (64 ) 

outline of the Proof for (62) j 

Applying the procedure followed to get (24), we 
shall have 


('<-a/p^)p'^ -2 -r “1 


ife((Xp..,Xj.)>/ H 


( 66 ) 
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where 0 < p^< i a^d ( Vpj^) + (i/p. ) = i. 


• • • »Xy/X-j^) =S 


^ X f(x^)(log /f (x2,.,.,Xy/x^)dS2, 

' \ S' 


. •-fill* 

« • « (^x^ ^ dx*! 


( 66 ) 


But 


log ; f (xg, .... ,'x/xj)^ 2 ‘ • • -dXj 

r 


'^'”1/ Pg 

= log r f ) 


I/P2 ^ ^ 

X f (x2/x2^)f (x^) • • • Xg ) 

^3+....+nj, 


dXg ... dXp 


dx. 


... (67) 

where 0 < pg < i. Choosing p^ so that (1/P2)+(1/P2)*1| 

applying Holder inequality and following the procedure 
adopted to get (24), we shall have from (67) 


Ho(( 4 ,....,X/x^)> H 


('<-l/P2)P2 


,tV%) 1 




Sp. 


2 


0( , 

(log X f (X3, .. ,,Xj.yx^,X2)cK3.. .dXj,)dX2 


®n3+.,.+np 


.... ( 68 ) 


Multiplying (68) by f(x-), Integrating over IL and 
making use of (66) , we shall have 



B|3^(X2> • • • >Xp/Xj^) ^ 


H 

C°«-VP2)P2’ 


(Xg/Xl) 


+ ^(X3,...,Xj/Xi, X 2 ) (69) 

Continuing the operation of getting (69) for 
another r-3 times, we shall have (60). 

( 61 ), (62), (63) and (64) can similarly be proved 
by repeating r -1 times the procedure adopted for getting 
respectively (30), (3i), (33) and (60). 

3.1 A Corollary 

If the vector variables ....>Xy are normally 

distributed then for «<> 0 

H^(X3^»-.|^) ^ * H^(X2) + ....+ It((^) (70) 

Proof of (70) follows from (64) by using inequalities 
of the form (52). 

distributed normally we shall have 

I^(X) ^ E^^CXj^) + ^(Xg) ♦ + H^(Xj^) (71) 

It may be noted that in the next chapter it will be 
shown that (7i) is not true in general. 

4. INPDIMAIIO MS MD THEI R PH3PEBHES 

As in part I for a univariate case, it can be shoMi 
that I|^(X) does not have the three basic properties of being 
finite, positive and Ijnvariant under linear transformations. 


From (70) in particular for a vector X = 



In. spite of this it lias importance since the °c- informations 
defined below depend on the difference of two such =<- 
entropies. 

We may define °<“information, ^ 1 , 

conveyed about X by Y as 

I^cCX ;Y) s ^(x) - l^(x /Y) (72) 

and =<-information, 0, «c i, conveyed about Y by X as 
I^(Y ; X) = i^(Y) - i(^(Y/X) (73) 

Note that in general I^(X; f) I^Cfj X) 

We may term I^(X; 'y) and I^( f 1 ) as <=<-lnforma“ 
tions of type I and type II respectively. 

In the limit, as °c - 1, both I^<X ; Y) and I^(i;X) 
tend to I^(XjY) , transinformation for such channels. 

4.1 Positiveness of I^ ( X; Y) and I^ ( Y; X) 

This follows from (ig) and (18) . 

Ij^CXj Y) * Io((^; X) » 0 if 1 and Y are independent. 

4.2 glniteness of I ^( X; Y) and I f^(Y;X) 

As shorn in section (4.2) of part I when for a 
univariate continuous scheae described by a random variable 
X, ^ 3 ((X) was approximated as a limiting case of the corres- 
ponding expression for ttie discrete case there appeared a 
term 

- Lt log Ax 
A X '*■0 
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that let Us to say that ^j^Cx) can be infinitely large, 
so also if we approxiiaate ^^{1) as a limiting ease of 
the corresponding expression when 2 takes only a finite 
number of values there would appear a term 


- Lt log AX 

-4 X 0 


(7MJ) 


where ax » ax^. .... ax^ , that can lead us to say that 
^ 3 ((X) can be infinitely large. But when an expression for 
IgjCX 5 f) or X) is approximated in the same way 

(m± 

there would appear two same terms of the type given 11*74), 
one with the positive sign and the other with the negative 
signj they would cancel each other and thus the resulting 
expression for I^(X j f) or j X> woiild generally be 

finite. 

4.3. Invarignce of ; Y) and I^(Y ; X) 

Under a non- singular linear transformation both 
I^(X i Y) and I^(t; X) are invariant. Let X and Y be 
transfoimed respectively to x’ and t by the non-singular 
transfoimations 


X * A X 


and Y* * B Y 


(76) 


subjected to these transformations we shall have 


H (l’) » 1^(1) '►.log Ui 


(76) 


and 


* \CS/ Y) ♦ log 


(77) 
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Using (?6) and (7?) m hare 

*• ViVf*) • J^Cm) (78) 

Similarly subjected to (75) we shall have 

^(I’) - I|^(Y»/Xl * Hk(Y) - ^<(Y/X) (79) 


°^“entropy of an ensemble has the same numerical 
Value when expressed in terms of either its frequency 
sampling values or time sampling values. 

Prooft In ippendix ¥I Goldman (3963) it has been shown 
that the transformation betw^n the time and frequency 
sampling values can be made a linear measure preserving 
transformation and we know that under any such transforma- 
tion the Jacobian of transformation is 1. So by (76) the 
result follows* 

Another possible way of defining conditional 

entropy H (t/X) is the following : 

Instead of taking the mean value of ^(Vx) the 

way we did for getting (6) from (6), we may consider again 

the strictly monotonic and continuous vector function 
(l*-®f) X 

g (x) *2 take the generalised 

mean of the 'information contained' variable H^(f/x) 

w.r.t gi^(X) as we did for getting (2). If we resort to 



■fchi s we shall have 


H^( Y/i) . g-^ I f(s) 2^’''°'’ 




dX) 


Using (5), (80) reduces to 


(80) 


^ 1 

B^(r/x) . — log^ ; f(x) { ; f'cy/s) dy) dx (8i) 

'<> 0 , °{ 4 1 » 

Jbllowing the sapie procedure as used for getting (8i) 
we shall have 

S (I/f) a —^logo ; f(f) ( X 6x ) d^ (82) 

1- *< V \ 

=<> 0, =( f 1 

In the limit j as •=< *♦ 1, (8l) and (82) tend to the 
corresponding Shannon’ s definitions. 

Comparisons of “C-informations defined on the 
basis of (6), (7) and (8l), (82) can, in the ssPe manner 
as in, section 7 of part 1, be made by using Jensen’s 
inequality for multiple integrals. 

Results, which involve specifically Ra^CY/X) and 
I( 3 ((X/Y) and given by the following equations, are also if 
we Use the possible alternative definitions of ^,((f/X) 
and given respectively by (8i) and (82) t 

(18), (lB)f (30), (33), (46), (49), (50), (5i), 

(62), (61), (63), (64), (78) and (79). 

On the whole all the results, except those given 



fey (24) and (3i) and their generalisations given respecti- 
vely by (60) and (62) y are true if or ^rso these alternative 
definitions of and . 

Using these alternative definitions all the results 
eniaaerated above can be easily proved by siiaple manipula- 
tions of the procedures adopted for proving them by using 
the definitions given by (6) and (7). For instance, the 
truth of result in (l8) for this alternative definition 
of i|^(y/x) can be argued as such t 

Jbr 0< «< < 1 , ctaaparing (8) and th© definitions of 
Ua((Y/X) given by (6) and (81) and retracing the steps used 
for obtaining (lO) from (8), the result follows : 

For 0 , by Jensen’s inequality for multiple 

integrals 


log f fCx) ( X f^(yA)dy )dx > X f(x) (log X (yA)dy)dx 

and further retracing the steps used for getting (16) from 
(ll) we shall have the result. Or else 


fl/ X*-***^^ C3( 

log X X f(x)f (y/^)dje dy * log X f (y)( X f (x)f (x/f)dx)dy 

... (83) 


By the argumwt used for getting (16) froa (13) we shall 


have 


X f 
®n 


l-<< q( 

it) f Wf) m 


1 
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Using (84) I (83) reduces to 

I { dx dy ^ log ; Ay) dy (85) 

% 

Multiplying (86) ly (l/l*«c) , a negative number, VB sb^ll 
have the result* 
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In part II of chapter 6 the results (18) and (19) 
and the invalidity of 


I^(X,Y) * ^(i)+ ^(Y/i) a ]aL<(Y) + > 0, oc=|r 1 

in general^ as shown t)y an ^aiaple in section 3.3 of part I 
of chapter S for a bivariate case^made us define two types 
of ®t-infoi^atlons. these are given by (72) and (73) in 
part II, chapter 6, two types of o<-infoifflatlons in a 
natural way give rise to two types of =<-capacities. these 
are defined in section 2. to evaluate °c-capaclties we need 
the maximum value of l^(^) or I[^('?) • As pointed|in section 
4 of part Hand proved in section 4.2 of part I for the 
univariate case, ^^^(X) can be infinitely large so this makes 
the maximisation of Iij((X) itself a proKlem. this has been 
tackled in section 3 for three types of constraints on X. 

In section 4 the invalidity of Shapnon’s results 2^(X) 4 : 
H^(Xj^)+..,. 4“ in gmeral for *<>0, ^ ^ 1 is derived. 

In section S.l evaluation of “C-lnfoimations ^en both X and 
Y are Gaussian and in section 5.2 evaluations of =<-capacifcles 
under two sets of constraints are dealt with. A generalisation 
of Shannon-Hartley channel- capacity formula is given in 
section 6. Ihe validity of these results is examined in 
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section 7 for the alternative definition of conditional 
^c-entropy, All the resialts tend in the limit as ■=< •* 1 
to the corre ponding availahle results. 

2* gLr CAPAGimsg 

Two types of ■=<- informations defined ty (72) and 
(73) in part II, chapter 5, give rise to two types of 
capacities ^ 0, «< 1 i.e» ®<»capacity of type I as the 

maximiM of ^-infoimation of type I and '<-capacity of type II 
as the maximum of '=<- information of type II. The maxiimisa- 
tion is to be done over the ybol& class of admissible 
input density futictionSf Symbolically 


Cl) r 

» max I^CX; Y) xs max 
^ f(X) ^ f(x) 




( 1 ) 


and 


( 2 ) 




(l) (2) 

In the limit, as *< *• 1, both and tend to C^, 

Shannon's definition of capacity for such channels. Since 

(l) 

in general Y) ^ X) , so in general 4= 

J2) 


3, MATtMISATIOM OF g^ Cll 

Pbllowing are the three sets of constraints on 
X for which the problem of maximisation of ^(X) is 


considered* 

(1) When the domain of definition of X is 
bounded i.e. < bp,.,., . 
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(ii) when the components of X 

assuae only non-negative values and have specified 
moments a^ia^t****) aj,^ Ca^i ag)****) ^ 

(iii) when the components %»****»^ ^ have 

respective specified standard deviations 

31scussion _of_Casg-.,ii). . , 

The problem is to maximise — logj ...X ^ (x)dx^ 

1 

.... .dx^ i.e. 

n 

maximise X^..f^f^(x) dX3^...,dXn for 0< «( < 1 
\ ®n 

and min» X •»..X f (^) dXj^.«..dx^ for 1» i»e« 

\ % 


opt, X^-.X^f‘^Cx)dx^...dXj^ subject to 
^1 ®n 

b_ ^ > 

X }.. f f(t) dxj^. . . .dxja - 1 C3) 

% 


m Know by the teclxaiqu® of Lagrangiah multipliers 

that the optimisation of X ^.--X ^f^(X)dX;L* subject 

^1 

to (3) must satisfy 


JL. * 
df 


>. l£ = 0 

af 


i.e. f ® 


lA-l 
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But the condition (3) implies that 




Hence max. ^(x) subject to (3) is 


log 




• • • • ®n^ 


same as the kno^ result for Shannon's entropy 

for X. 

Jai; acBs.stofi o£,.ff,ase Cii). 


The problem is to maximise 

°° °° at 

X- f dx-,...dXjj subject to 

I** ^ 0 0 


C» <» . V 

X X f( 3 :) dx,.....dxn * 1 ( 4 ) 

0 ^ 

and 


OO 00 

X,.. X f( 3 £) dx^*...dXn = 
o 0 ^ 

OO OO 


X....X Xo f(X) “ S 2 



00 OO 

X...4 X f(x) dx^ dXji * an J 

o 0 


By the techniqu® of lagrangian multipliers ne know that 

OO OO 

f(x) opttolsiBg ^ * * * ''^'^ to (4/ 

0 0 ^ 

Qjod ^5^1 ffiiist 





Oi 

if * 


i.e. f(x) s ( — ^ 2 2- ) 


( 6 ) 


Using (6), (4) reduces to 


oo 




o 


oo -A -Aix,-. 

..; ( 1_ 

o « 



_1 

< -1 

) ex^ 


. . . dx 


n 


1 


(7) 


In (7) for the integrand to be positive we assume A , t 

...... to be negative. Integrating (7) first for , 

then for Xg and lastly for ^ it will be found that it is 
convergent iff < «( < 1 and it would ultimately 

reduce^ to 


3/oc-l 
(!/«<) (-A) 


1 

<f n. 

c(-l ** 


^ +n ) 


= 1 


( 8 ) 


Similarly, using (6), every meober of the set <6) 

will be convergent iff < 1 and the set itself would 

m- 1 

respectively reduce to 


(-A) 


a(-i 


^ X 


.(-^+ n + 1) 




oc-- 
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31“ + n + 1 


«C-1 




oC-l 






ag' 


^1 }^ji • • • )vy\ ( 


*<-1 


M-l 


=<-l 




+ n + 1 


i-H 


= an<«t) 


0(-l 


1 1 

A? Ai - An 





... (9) 


Dividing (8) by the respective members of (9) we 


shall have 



1 

°c-l 


+ n + l) 





- >A ( *► n + 1 ) * 

■jr^ o(.i 





1) 



y 


CIO) 


MTHttplyln* t>“ 


n 

JL + K**! ^ 

( • • • * ®n.^ ^ ®t*l 


n 

i-h) 

>i, . . • . ^ M 


( 11 ) 



So using (6) m laave 


00 oo 


Q o 0( 


->nXa 

a )d*i..dx„ 


... (13) 


But 


oo oo -yW-A,x -AnX_ 

/...f ( 1 2 

O O 0( 


oC 

1 


) dx ....dXjj 


oc/°t-l + n 

(V'O (■•^) 


C ^ n) 

oC— 1 


■>. 

< 


i’ i 

<=(-1 




+ n 


(8)J 


(13> 


Comparing (8) and (ll) 


IH 2) o) 




!•< 


- ^ X ( 


(14) 


Using (14), (13) reduces to 
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“ f°°, ■ ^ *r . 5 :^ 


f • • • X ( 
o 0 




n(l-«()-l 

Ca^. * * ■ a^) (n(o(-i) + «<) «5( 

“ll** 

(pc^ - n+l)/ 


(16) 


Using (l5)| ( 12 ) reduces to 

max H^(X) » log (a^. .. 3 ^^)+ n log |^nC<<-l) + 
1 n(°(-i)'»^ r 


«< 


• log 


°C( 2 ®<* 1 ). • - (npt-n+l) ( 16 ) 


P < «( < 1 
IM-1 


In the limit, as 1, (16) tends to 


Ufc. * 

max. H.]l(X) » log (a^ ♦ p 


log a^^e + log ag® •*..+log a^^e 


as already known. 


sous si ^ 


supposing E(%) « 0 is? l,2,....,n, the prohl®m 


is to maximis® 


00 00 


i_ log / ...; t l.e. 

mOQ •iC® 


00 <30 


**“ 


( 17 ) 
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subject to 


00 oo 

ni) dx,. 

.00 »oo 1 


■'**n “ 1 


(IS) 


OO OO 

J •••X f(x) dx-....<iXn = 

-OO «oo 1 1 u 


oo OO 

x^ f(x) (3x.,.,,dx_ = 
.00 .00 n 1 ^ 


(19) 


By the technique of Lagrangian multipliers we Imow 

oo oo «r 

that f(x)j, optimising X...| f^'Cx) dx. ....dx- subject to 

-00 -oo 1 “ 

(18) and (19), must satisfy 


^ * -A-ii j 5_ <*? « - 

df af a f ^ 


,->viJL- (x*® f) « 0 
a f ^ - 


i.e. fCX) 


^ 1 X + 


.+ ^ x_ ‘<•1 

) 


( 20 ) 


Using (20), (18) reduces to 


o(. 2 . oo oo 2 -'-x 

2^(lA) X*.4 (^ + A^ Anx^ ) dx ...dx^s 1 (2i) 


0 o 


The integral iu (2l) is convergent iff — - n + 1>1 


< «C < 1 • lo Integrate (2l) over x for fixed 

values of x^, we «-rst malce the substitution 



-170 - 


Xg 4 ) + A| X? SI t 


then 


/^ + Aa Xg + , , . ,+ Avt 


* 1 


in the result so obtained. If we siiiip!}J.fy further, we shall 


have 


n-1 °c-l T «c + l 1^00°® « o 2(=<-i) 

2 (!/'<) y~“"' P ^2 Tx-5T * 2 • • XC/* ^ 


o o 


dXg. • .dx^ — 1 


(22) 


Repeating the above process for Xg,...,x^ we shall 
ultimately have 


l/°^— 1 - 

( !/■<) ' ^ 

V An 


£ 


.”<-1 ® «(— - ^ 1 1 

r i ’ i 




(a.3>) 


Osing C^) in the respective members of tbe set of 

equations (19) and maicing use of the procedure adopted to 

get (23) from (2l), we shall find that every member of the 

n 

set (19) is convergent iff < =< < 1 and the members of 


(19) would respectively reduce to 


3p(-l ^ 

L 2('<-l) 2 2 

^ cl' 3/g 2(l-«) 2<l-<) 2 2 

A, y 2:2 2:i 1 ) = °T 

••••^' 2(i-«r 2*2 , 



1- ^ 4 azi 

1 ,”‘-1 3 3 . .1 1 - 




.% ■ 

>.y\ 


• • • ♦ PC 


— .Sll 

2(l-o() 2 


, 1 ) s CT^' 


l/=<-l 


3<-l 


(V'<> 




2(o(-l> 


I3i=l 
+ 2 


Vsu _ 

^'yi V >^1 Avi- 


a ^ . 1 


^‘2(l-«)’ 8 


) p(^-:-,- 1 , i )i 

2(l-c<) 2 2 


.P( 


> 


3=c-l n-1 1 

T T ^ * o ^ (24) 

2Cl-<) 2 2 


Dividing (23) wltii the respective memhers of (24) 
we shall iJave 


>1 n(“C-l) 4 2< 1 

^ 1 - «( 07^ 

(26) 

, n(«C-l) + 2=( 1 

Z' X - c( 


Multiplying the m^ihers of (26) we shall have 



X 


CTr, 


n 



n(^^l) + 2°( 
1-eC 




/S, > v\ 


(26) 


Using (20), (l7) reduces to 
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max. Bt((X) * 


-^10, 

mOO woo €< 1 


ax (27) 
n 


where , Ai are given by (23) and (34:). But applying 

the procedure adopted for getting (23) from (2l), we have 


^OO mOO OC 


<Vo(-l 

) dx^. . 


. . ^'^“1 A “<-1 o 

:(1/^) A 1 2 


o( 

P (l-«< 


^ X % of 

2^2 -» ?L 1 


^1 An 


.. (28) 


Using (23), (28) reduces to 


foo":io^ 5 ' 


.OO .mOO 


~ ^^1^ " g"* i ^ 3^( 3/2) 

o( p( l/i.o( -i/2,l/2).,.p(l/i-c< « n/2,1/2) 


(29) 


Comparing (23) and (26) we shall have 


2/‘<-l n('<-l)+3=< “a a 
(1/«C) C — ) (^ ) 




n - 1 

5 » 2 )) / (30) 
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Using (30) in (29) and simpli lying we shall have 


oo OO ^ 

/ ...X C 3 ^ X 

-oo -OO QC * 


) dx^. ...dx^ 
g(-^)-1 


- 1 ’I ’-ocW'l’il 


(1- «c) 


mi«o( ) 
2 


Using (3i), (a?) reduces to 


1 Ini 1 

max VX) =. — log— — _ + log 3(_ - i, i )..p(— - | 1 ) 

n(°c-l)+ 3a( l-oc 2 2 ^ 5 


* ^ lGg(l-°() ♦ |log n(o(-l)+2^ -►logC®! (32) 


n+ 2 


<<K <1 


Lt (l/l-«<) log 

o( 


n(«(-l) + 2 «c 


(33) 


log Tm ^ ^ . i ) 


n 1 1 
2 ’ 2 ) 


1 1 
» Lt log M — • 1/2, 1/2)+ ...+Lt logP(-^ 

0 ( -*-1 "<*1 o ( 1 “* 


, 1 X 

l-« 2 2 ^ 

... (34) 


By Ganss definition of Gapuna function, page 770, Kreyszig (1962), 

r( 

we have ' 


P a( a Lt 

I V . ->Q 


n 1 'll 


«<(«<*^l)(®t*'2), . . . (o<fn) 


«<> 0 
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Using this (34) would reduce to 


L t log 

o( ^ 


H 


i-«( 


i » i ) gz-JL - n 


^<i=sr *“ 2f > o ) 


« Lt log (i-<> + n log /^i 

a( 


(3S) 


Using (33)} (34) and (3£)} in the limit, as 1,(32) tends 


to 


max H^(X) « (n/2)+(n/2)log 2 + log(^ --on ) 4 ^ n logAr 
* log 2re ^4' log / 2ire 

as already known under the given constraints on 

4 . i Mm imi 

It was pointed out in section 3.1, part II, 
chapter 6, that 

B^(X) ^ I|<(%)+ + 0, ^ 1 (36) 

is not true in general* The following is its proof. 

E£SSX* 

let us consider the result given by (32), This 
gives the maximum value of It((X) when each component of 
has mean zero and a specified standard deviation. Had (36) 
he^ true in general we would have :^(X) to be maximaa 

when 

(!) all the dimeasions are indepen^lfiA^ 
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(il) each dimension has the greatest “C-entropy 
under the constraints, 

and the result lould have 

max Et<(X) as n ["(l/l-°f51ogC2o</3o<“l)+(i/2)log(3=<-l)- 

(l/2)logCl-«()+logp(2(2^ > l/2)] 

-► log («T ) (37) 

since max. hy putting n asi in (32) is 

1 ^ 1 1 '^’^1 1 

3^1 * 2 a lo8(l-<)+logp(g^f^)> 2 )+log=ri 

for 1/3 < « < 1 

t/t" 

A comparison of (32) and (37) shows that^is not 
so. Hence we can say (36) is not true in general at least 
for (n/n+2) < «< < l# Hence it is not truej^for all '<»*</' 0, 

5 . n mMmMs 

In 6,1 “t-infoimations and in 6.2 ®c*capacities 
for specific multidimensionai continuous channels are 
evaluated. 

^•1 °<-Informati ons When 1 and j are gaussian 

From section 2.4, part II, chapter 6 we have 
for such a case for 0, 1, 

^^0:) - » ^(x) - 


and in the limit as 1 





%(Y) - s 

and this >s taiom to be true. So for =( > o, ^ 1 
} t) = X) * 1^(1, Y) 

and we know from Geffand and Lagloia (1959) that 


Ij^CX, Y) * 



det A det B 
det C 


where A» B and C are respectively the moment matrices of 
the distributions of l and their join. 


5.2 °("CaDaclties 

In (i) expression for '^-capacity of type II is 
derived when the channel is affected by additive Gaussian 
noise and each Aperient of the input vector is distributed 
with mean zero and a specified standard deviation and in 
(ii) ®(- capacities will be considered when besides conditions 
in (i)| input is also Gaussian. 

(1) Let the input be represented by the random 
vector X * p^l^*-** output by Y a Y^J and 

noi se by Z * ** ** ^ • 

The assimptions are * 

1. S(ljc) = 0, 1(4) « °3ic , k • 

8 . !{;%) = 0 , 1 ( 4 ) • =^4 > ^ 


and 

k*l 


exp. 


(-% ^ 



«/*2ir 
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3. f a X + 2 

By assmptioiis 2 and 3 

f(y /x) X f(x + £ / x) X ^(z) 

Using this we have 

Hence, uging this for *<> 0, ^ 1 , 

I^<Y; X) a H^(y) - I^(f/x) a ^^(y) - 2^(2) (38) 

Using assumption 2 we shall have 

^ n n 1 

^<(Z) » 2 logC/‘2r cr^ ) + Xog “ (39) 

lt*l 2(1- ci) < 


Using (2), (38) and (39) we have 

(2) „rn n i'l 

a max I^(y;X) a max ^(y) - 2 logTar + log i 

^ k=l ^ 2(l-o() < j 

.... (40) 

Since each component of X is affected by an iniependeiit 
Gaussian perturbation i.e. 

% • ^ > 

'|according to this and assumption 1, Yjj. is distributed with 

mean zero and standard deviation ) , So the 
problma of maximising ^(Y) conforms to the problem discussed 
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a* cas. (lii) in 3, 

reduces to 


( 2 ) 


1 

l-«( 


2 oC 


— ■*■ loe(S(-L- - i 


* *** \ 

2 * 2 '*** 


.»( 


1-oC 


n 

i 


’ f >) • 7 log(i-o(>+ ^ log(a(°<-i)+2oc) 

iS .«S 2 


a 


2(l-o() 


log < 


+ log /■( . / log (41) 

(a/n+2) < «< < 1 


Using (33) , (34) aad (35) the expressioa ia the bracket oa 
tile right haad side of (4i) beads to (n/2) log 2 + a logvAr 
So in the limit, as «C *• 1, (4i) beads to 


B 2 2 

* (1/2) 2 log (1 + OXj^ / a-^ ) 

k*l *• 

as already knowa. 

(ii) la this case the st^ps for gettiag the result 
given in (40) are the same as those ia case (i) above. Since 
the input is also GaUssian, so X is normally distributed 

<a A 

witla * oxj^ ♦ I ^ * l»**»i Hence by (7i) 

ot part II, chapter 5, ^,((f) is maximum when (1) all 

the dimensions of X are independent random variables and 
(2> each dim«nslon has the greatest «<*entropy} but Xjj;*s are 
normally distributed with mean zero and standard deviation 

2 2 

^ -fr 


Zjj,)- 


So cad-cuXations would give 
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* log/2ir( ) + L log i 

2C1-OC) 

and hence 

«az H^(5) logyar( i^) -a^jlog i (42) 

Using (42), (40) reduces to 

(2) n cf^xv 

Got » - S log ( 11' « ) o( > 0, a( fs 1 (43) 

® k»l 

SlBttilarly we shall have 


1 a 

“2 log ( 1 4- 

ICJsl 




) oC>0, ^ 1 (44) 


(l) 

Hence In this case both and («< turn out to he same 

as the known value of corresponding Shannon’ s capacity. 



If we consider a class of signals represented by 
time functions f(t) which are hand limited in the range - 
to + cycles per second and if we further assimie that these 
si^aals are also time limited i*e. f(t) is negligihle outside 
a time interval (-f /2 § 1/2) , T being an integer, then we 
know that such signals with their power content eqpial to a 
preassigned nmber are represented by points in a 2«^T- 
dimensionaX space* 
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How a point in this signal space may he put as 


Similarly the output signals may he represented 
by points belonging to another 2w X- dimensional space. 

So a point from this output signal space may be put as 

i - Ig, JSVT 

So the study of the transmission of band* and 
time- limited signals tantamount s to the study of the 
behaviour of a 4w X-dimensional random variable Z * ^X, 
Proceeding exactly the adopted for getting (2)^part II, 
chapter 5? and following the manner of section 2.1, Part II, 
chapter 5 , we can define various “^-entropies connected with 
this 4w T*dimensional random variable Z. Hence under the 
conditions subject to which (4i) is derived we shall have 
oc-capadty of type II, CwT)/(wX -f-l) < < < 1, for band-and 
time- limited signals i.e., 


( 2 ) 




wf (!-<<) ♦ e< 




..pc 


1 

1*«( 


2 2 

- wT, 1/2) ) 
wl 


wT 

- w f logCl-<) + V Xlog (2w T<‘<-l)+2=< )» — - log oC 

1-«C J 


2wX 

<(- Z log 
k*l 


,4 


2 2wl ^ 

cr-TSt • 2 logVOT 

^ k»l 


(4S) 


(wT /wt +1) < *< < 1 . 
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In particular when 


2 2 
te crx 


\ 


J/ 


^ s ij * • « • ^ n 


2 

<y-2 


(46) reduces to 


,(2) fl *< ^ T /er ^ 1 1 'i 

■— log + logC^C— - - , !>••.* 

I'^t wT( l-«) ♦ oc l-°< 2 2 

.... - w T , i ) ) 

l-«< 2 

wT 1 

- ¥ Tlog (l-oC) + wT log <2¥T(1-°<) + 2o<) + log oC I 

1*< ^ 


+ w Tlog( • ¥ t log^ T ) (46) 


<0C <1 

wlM-l 


Using (33), (34) and (36) it can be easily verified that in 
the limit, as 1» (46) tends to a known form of Shannon- 
Hartley foimula* 

Under the conditions for which case (ii) in 
section 6.2 has been tackled it can be easily verified 
that both and would turn out to be same as a 

known form of Shannon- Hartley formula under these conditions. 



7. DEFII^mOH OF COaDIgOS/a. 


As developed, in section 5, part II » chapter 6, 
th€A 0 are alternative definitions of conditional °<“eatropy 
and equivocation for multidimensional continuous channels* 
These are respectively given ly (8i), (82)-part II » chapter 5 
It may be noted that all the re suit St that Involve 
thee© concepts of conditional ®<^-entropy and «<-equi vocation 
and are developed in this ch^ter, are also true for these 
alternative definitions. This can be easily verified. 
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As defined in Felasteln (2956) a s®!Si"'<K>ntinuous 
meaoryless cbatmel Is a chanritl with input consisting of a 
finite set and output an infinite set with a prohahility 
measure defined on the output for every inputt so a multi- 
dimensional semi-oontinuoug memoryless channel is that for 
which the input consists of a finite set of n- tuples and 
output an infinite get of m-tijples with the further restric- 
tion that each transaitted n» tuple is independent of the 
previously transmitted n-ti^le i*e» no memory is involv«l$ 
however cc»ponents of a particular nf*tiq»le may he Interdepen- 
dent* I* t the Input consist of s n- titles 
let X • rx-,....t 3 l 1 he a random variable which takes on 

the value x ^ with pro bahllity p(r^^)> 0 so that^X^p(» ) 
« 1 • Let the output he descril^d by *? • 
h avin g the density function p(^) t positive and defined over 
an ffi-dimaasional rectangle % so that X pCf) ^ • ®*® 
case when pC^) vanishes iw most set of measure xero 

can be included either by adopting shani^m' s convengion that 
0 X «»* 0 or by evaluating the various ^-entropies associated 
with such a channel the extended real nmber system. As 
for a fflulUdlmenstonal continuous channel;, there are also five 
x-entropies associated with a chtfuaei under consideration. 
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( 1 ) 


Associated with the input 




1 -< ^*^§2 2 P ( * ) 

i«l 


o(> 0, o< 4 1 (1) 


H^(Y) 


Associated with the output 
1 


^82 X p‘^(y) dy «c> 0, « ^ 1 


l-0( 


( 2 ) 


(ill) 4ssociated with the syst^ as a whale 

I^(X| Y) a -i: — log V ; p‘^( y) dy (3) 

1 . oc 2 i ^ ^ 

(iv) Conditional “t-entropy ]|^(5/x). Since a 
transmitted vector can be received as any admissible 

value of Y and [ f(y/'2:^^^) dy = l , so following the 
procedure of getting (2) we shall have 

\(i/ logg / f'cs^ / df 


So for ^ 5 ((Y/X), it is, as in earlier chapter*, appropriate to 
take the matheBiatical expectation ^^{t/ Hence 

^XVt) * -i— S p(3t^^^ log 3 S f^if/ X^^^)dy (4) 

1 - « i \ 

(< ^ i 

(v) °c«equi vocation H^CS^Y) . Based on the procedure 
adopted for getting (4) we shall have 


1 of "i 

H (X/i) s X fif) ( logg s p Vy ) ) dy(5) 

“ i-« 1%. *■ 


K > iJf I 
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It Can be easily verified that in the limit, as 1, (1), 

(2), (3), (4) and (3) tend to the known entropies for such 
channels* 

All the other results proved in sections 2 and 3, 
part II, chapter 5 for multidimensional continuous maaoryless 
ch^nels can similarly be proved for such channels* The 
concepts of '^'“informations and «t“Capacities defined for 
multidimensional continuous channels can likewise be defined 
for such channels. 

In section 2 available Fano’s inequality for such 
channels is generalised. As a consequence of this generali"* 
sation, the converse of Shannon's second fundamental theorem 
for semi -continuous memory less channels is generalised in 
section 3. In section 4 these results are considered in the 
light of alternative definitions of conditional «<-entropy and 
oc-equi VO cation. 

The operations dealt with below are contingent upon 
the existence of underlying integrals and the application of 
** Jensen's inequality for multiple integrals” is made subject 
to the restrictions as stated for its validity in part II, 
chapter 5* For the rest the base of the logs is assumed to 
be e. 

2. GEKERALISATIQH QF_._JFAfiQ'_6_IjmJALlTy 

We know that for such channels Fano's inequality is 

given by 

^ H^(p(e), 1- p(e)) + p( 0 ) log (s-l) (6) 

where p(e), the probability of error of the code consisting 
Cl) 

of s words X , is given by 
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* 1 r p(gCy), f) dy (7) 

\ 

(7) is calculated on the assumption that the decoding scheme 
is such that assigns the vector g(^) = x'*'(say) to a received 
vector y • Ulth this (7) can be written as 


p(e) s 2 J f(x, f) df i 

% 

where stands for the set of s code words. From (8) 


we have 


1 • p(e) * / p(^ I f) dy 

% 

R)r =<-entropies (6) iidll get generalised to 


r«< 1 

H^(^) ^ (l/l-‘<)log p (e)(s-l) + Cl-P(e)) J 


o(> 0, x + 1, 


( 10 ) 


It Can be easily verified that in the limit, as 1, (10) 
tends to (6). 


^(X/l) given by (6) can be written as 


= (i/i-<} ^p<y> /y) <*y 


( 11 ) 





But by Jensey’s inequality for multiple inte grains*, page 2l, 
MoAxey (1966) we have 


X p(y) 


log i p'^C^y) 

. lx] 


df ^ log ! pCy) 2 P^Jx/y)+ 

^ P (X /y) dy 


J 


» log 


r X 4 )(y)p‘^(x/y)dy + 

X 

f p(y) p'^Cx* /y) df 


J 

s log [x 2 P*^ *(y)p‘^(x/y) p‘"(y) dy 

X 

* t p'^CxVy ) p^"^(f) dy 

Rm 


li 

J 


log 


+ ; p°‘(5’, y) 4?1 

J 


But by B>U«r Inequality for soma, page 21, Beckentech and 
Bellatfl (3365), 


B p“‘ .(i,y)p'**cy)« Si, i* ^ 

\X]-^ tX|-x 

IntegraUng (l3) over % 


l-oC 


(13) 


1“< 


t { 2 pV.y>f 
LSX!- i 

“ ... ( 14 ) 
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Applying Holder inequality for multiple integrals 
Beckenbach and Bellmgp (1966) to the R.H.S. of (l4), 
it reduces to 

of 1*^ 

‘ i (I, 5 < I (s-l) P<y> ay ) (15) 

\«i- ^ \ 


Again by Holder inequality for multiple integrals 


I t) p^‘ty)dy ^ ( ; p(X^ y) 0 (l6) 

\ , % 


Using 
/ pCy) 

^ log 


(16) and (16), (12) reduces to 


log I p‘^(^y) 

ixf 


dy 




1- oc 


(£ 1) P(3e,y) #) (s-i) H 


^ X 


I p(:^^y)<if)**^ (17) 

J 


Multiplying (1?) by (l/l-^) , a positive nxmber, and 
Using (8), (9) and (11) ve shall have (10) for 0< «< <1 . 

c^ftse Mhen <=< J> 1 
a( 

Since 5. p (x/f) 1 » so Jensen's inequalilgr for 

\xl 

multiple integrals 
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X pXJJdog s p‘<{Vf))dy> t pifx 5 p°‘(S/?)iog £/(i/y))aj 

\ (Xj ,Xj 



... ( 18 ) 


But ; p(f) (.Zp^cs/y)) dy^ 1 , let it be pi 


Using this ( 18 ) reduces to 


£ p(y)(log § p”^(3^f))dy P log : f p(y)( 2 .p*^(3c/y))dy ; 


* p log i I z » p(y) p*^<Vy) dy 

' \ 'Xj- 
*** 


♦ / p(T) p'^CxVy) dy 

-> 

» p log iX { 2 p^"^(y))dt 

j\ X*- 

oc ^ i-rfjT ”*n 

'*■ X p (x I y) p (T) dy j (19) 

w i 


But by lolder inequality for ^s 


^ "i ^ *1 '**QC 

2, ♦p‘^(iiir)p^ (f» <2, .^(x,y)) ((s-l)p(y)) (20) 

[Xj- S Xi- X 


Integrating (20) over \ and then applying Holder inequality 
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for multiple integrals to the B. H«S» of the result so 
obtained we shall have 


iCl. 1 X s 

JWW-x* ^iXI- 


p( 3 E,f) dj)°' 

£ 


1*^ 

( J (s-1) p(y) dy) 


0( 1^ 

* ( X s pC.i, 7) dy) (s-l) (21) 

%W- S* 

Again by Haider inequality for multiple integrals 

X p'^Xi’, y) p^'^(y) dy X < X p(i* I f) dy )'‘ ( 2 S) 

\ Sn 

Using (21) and (22), (19) reducds to 

X p(y) (log 2 D‘‘(i/f))dj, B logljx £ p(^,y)dy]. (s-l)’''°‘ 

^ [xf ^ 



Multiplying (23), by (3/l-<) , a negative number and using 
(8), (9) and (ll) and the fact that P ^ 1 we shall have 

(10) for !• 

This completes the proof. 
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3. OF TH E COHVSxliE OF mMQl Ui 

■SmM. yUIDAHMI AL 


A geaeralised form of the converse of shannon* ^ 
second fundamental theorem is stated and the proof 3- ^ 
gi¥en for a faMly of claaJinels consisting of tne ex 
sions of any seai*continuoxis memoryless channel suh5 ^ 
to the further restrictions that the input probatoil^ 
distribution is a product distribution and its cod© 
are selected with equal probability. For this ’we ^ \ 
following lemma. 

^emma s -^-capacity of either type of an nth order 
si on of a s^i“Continuous memoryl®ss channel wi'tli 

JBL g 

probatilil^ distritution as a product distribution 
times the «t-capacity of the corresponding type of 
channel. 

The proof for this lemma can be constructed. I**' 
same way as that of the corresponding lemma for di^^^ 
memoryless channelsj section 3. 2j chapter 4* 

Below the symbol (s, n) stands for a code 
with s input n“sequences. 

of The G eneralised Convensg. 

Given € > 0 , it is impossible to transmit »• ^ 

(s,n) lAtH B = through an nth extern ^ 

of a seml-oontlnuous n»ioryless noisy channel hawt**-® 
=c-capacity 4^’ * probaWUty of 
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desired. 

As “f *♦ 1, tile above state'sent tends to an 
already available statement. 

Using (lo) and ttie above lemma and following 
the lines of proof of the corresponding generalisation 
givaa for discrete menoryless channelst section 3.2, 
chapter 4, the proof of the above statement, with the 
additional assisaptions that the input probability . 
distribution of the nth extension is a product distribu- 
tion and all the s input n-sequences are equally likely, 
can easily be constructed. 



Following the procedures adopted for getting (8l) 
and (82), part II, chapter 5, the alternative definitions 
for conditional “<-entropy and ^^-equivocation for channels 
under consideration are given by 

4 1 

( 24 ) 

H (S/I) « (Vi-<)iog t p(y)(2 p°‘(J^^Vy)) dy ■=<;> 0, « y 1 

It may be noted that all the results developed 
in this chapter are also true for these alternative 
definitions. Ibr instance, the truth of (10) for ^(J/X) 
given by (2i) can be seen as follows t 


H (Y/X) a (1/I^)l0g 2 p(x 
■=< i 


^(i) 
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■Case Mhen 0 ^ of ^ 1 

As in (12) we have 

log ; p(y)r| * log 


S 2 p*^(x,y) p^(f ) df 

BjjlXj- X* 


+ f p‘^(X^ f) p^’^Cf ) dy 


Using (16) and (16), this reduces to 


log ; P(y)r2 D^CS/^ldf ^ log 

iVi L{xj J 


of 1 *™®( 

(f 2 p(2^^,y) dy) (s-l) 


H t p(^% y) dy) 


( 26 ) 


Multiplying (26) by (Vl*®^)* a positive number, and using 
(8), (9) and (11) we shall have (10) for this case. 


gas§ Ifeea 5. 2 -i 

Using (21) and (22), (24) reduces to 


log S p(y) 
"n 


Z Si*(.i/S) 


(■ 


©C X""®C 

dy> iogj( ; £ ^s"i^ 


+( / p(±^, y) djf 

\ 


( 26 ) 


Multiplying (26) by (3/1*^) > a negative number, and using (8), 
(9) and (H) we shall have (lO) for this case. 
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fissms. 

^Y,-LIE2KIC QEkMELS 

immmuQ^ 

Ibis chapter is also divided iato two parts* 

Part I deals with the definitions of °c-entropies associated 
with different steps of a Morkov chain and s(a®e relations 
between them. These relations tend in the limit as «< ■* 1 
to the available relation connecting shannon’s entropies 
associated with different steps* In part II “^-entropies 
associated with a stationary source and basic concepts like 
rates of transmission of informations through a stationary 
symmetric channel are developed. 

£MI— I 

1* 5TA:riQlABl,TY.* mimOlS ■ 

In section 1*1 concept of stationarity of a Markov 

chain is defined and some notations are given. In section 

1,2 o(*entropies associated with different steps are given. 

1*1 Stationarity And SotaJ^ofla 

Suppose a sequence of trials is performed and the 

resxilt of each is n leutually exclusive events? let, in 

particular, the sth trial result in n mutually exclusive 

events A homogeneous or stationary 

Markov chain is one in which the conditional probability of 

(s+i) 

occurrence of the event A± in the (s+l)th trial, given 

C s ) 

that the event % has been realized in the sth trial, is 
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Independent of the number of the trial } the probability 
is called the probability of transition and is usually 
denoted by probability of going from the ith 

event to the 4 th evmt. 

Below these events will be named as states and p^^ 
will then be interpreted as the transition probability of 
going frcm the ith state to the Jth state. Since our 
dealings are to be confined to stationary Harkov chains* 
so it is sufficient to denote the mutually exclusive events 
associated with it* called states from onward, without the 

superscript (s) i.e., by simply Ag, 

1.2 ±:BHLtr9B^8 

Let Ap*..., be the finite number n of states 
of a chain and let the matrix 


^11 

Pl2 

Pin 

^*21 

P22 

- - - P2n 

i^nl 

Pn2 

■ Pm 


where Pij 
n 

and S p^.*l,i* 1|2, ..*..n 
4*1 


be its transition probability matrix 


( 1 ) 


By (l) » the subset of transition probabilities 
from the state A^ foim a complete distribution, so we can 
define on#*step “(-entropy, to be denoted by , 
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Cl) 

%o( = Cl/l-°() 


n 

log 2 
j=l 


0< 



y 0, <=< f 1 . 


If the probability of the system being 
initially in state is designated by i.e, if 

^Pl» specified set of initial state 

probabilitiesj so “(-entropy or “(-uncertainty of the 
chain should naturally be defined as the average of 
Hj^oc i«e. the “(-entropy of moving one step ahead from 
any initial state. So denoting it ty » we have 


(1) n c( 

H. = 2 p, ( l/l-o( log S p>. ) 

« 1=1 1 3=1 13 


or 

H =(l/l-“() 2 p= log 2 p= . , (2) 

«( i=l ^ 3=1 

“(> 0 , “( ^ 1 . 

( 1 ) 

We may call » the “(-entropy of the chain 
with specified initial probabilities for moving one step, 
In general, the set of events of going from 
to any other state in r steps i.e,, 


' (r) (r) (r) 

/AjL » Ag /Aj^f ***•••••••• ) ^ 

constitutes a finite probability scheme with n events. 
The r-step “(-entropy of this finite scheme is, denoting 

(r) 

it by , 
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H 


(r) 


\ 

= (1/1-eC) log 2 (pf' ) «{> 0, ^ 1 

<13 


where ^ 0 is the probability of moving from ith to 
the jth state in r*“steps • Hence the r~step •entropy 
is 

Jr) » (r) 

X * Pi 


f / ^ n ^ , (r).“C 

H » 2 P4 log 2 CPi4 ) 

^ ial ^ Jsl 


>0, oc ^ 1 


where 


[pi. 


'^n 


(3) 


is the specified set of initial 
**J 

probabilities. 

In the limit as 1, (2) and (3) tend to the corresponding 
Shannon's definitions • 


2. RH, A'goaa Id -^-MTiPPiia 

Sd&e relations between “^-entropies associated with 
different steps are given in the foim of two theoroQs since 
the results of theorem i are true only for regular Markov 
chains, so we shall enunciate these terms in section 2.1 and 
the theorems would be given in section 2.2. 

2.1 ^nown Results 

(1) We define a chain with transition probability 
matrix P a regiaar chain if there exists a positive integer 
r such that F ^ has only strictly positive elements and 

(2) the steady state pro bablli tic* arise if for seme 
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n, may be very large, P ^ has almost identical rows i.e., 
the prob. of being in any state s^(say) at any time t= n 
(or after n-steps) is almost independent of the starting 
state i.e. at t = 0. Mathematically we can estpress this 
condition by saying that 
(n) 

I't Pfi * Cl> i * 1}2, n) 

«♦ 00 •' • 


where wj is independent of i. The nmber wj is called 

the steady state probability of the state s^ . So in a 

few words if there is a set of numbers W;^^,*..., such 
(n) 

that * Wj (i,j ss 1,2,...., n) then we say that 

steady state probabilities exist for the chain. 

(3) Another result that we is tha j 

following theorao, page 174, Ash (1966), 

’ 'If a Markov chain with transition matrix P has 

steady state probabilities w. (j s!l,2,. . , . ,n) then 
n ^ 

(i) S Wj * 1 (ii) W* fwj^, ...., is a ''stationary 

J«1 " ^ ^ 

distribution*’ for the chain i.e. 5fiP * ¥ (iii) this 

stationary distribution W is wiique . 


Th.l If a chain has all ^ 0 

probability matrix and jp^, ...., Pj^^ 
probabilities then for 0< «< < 1. 



( 1 ) 
r H 

(«<- i/k)k' 


in its transition 
are its steady state 


where 0< k< 1 and k* is so chosen that (lA) + (lA*)« 1 
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(ii) ^ ^ r max 

and for 1 

(ill) ^ r H 

(“C-lA)!!:' 

where k and k* are chosen as above 

<iv) r min 

1 


Hotet that the results (i) and (iii) are infinite sets 

of inequalities each member of either tending in the 

(r) (l) 

limit* as ®c ”• 1, to the known equality = r • 

As *< *• 1, results (ii) and (Iv) tend respectively to 

(r) (l) C"^) 

^ r ma?: and ^ r adn. . 


tmit 

Since y 0 , so the chain is regular aJid it 
is given that Pj^ are steady state probabilities, hence 
by result (3) stated in section 2.1, we l^ve 

h ’jfih “ii- ^ “ 


(1) the result is trivially true for r 



(«c-l/k)k* 


1 l.e. 
( 6 ) 


the equality will occur when p^j are equal. (S) lays 
down the basis for induction. Hence suppose the result 
is true for some r ^ 1 i.e. 
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(r) 


T H 


M) 




we shall prove that it is true for r+i i.e. 


( 6 ) 


2?+l /,) 

Bk (r^i) 


(7) 


let the system he in the state ^ , The finite 
sch^e which describes the path of the syst^ in the next 

(r+1) steps can be regarded as the product of two dependent 
schemes* 

X t the sch^e corresponding to the immediately 

/ •! \ 

following step with the “t-entropy H ^ and 

Y s the scheme describing the system in the next r 

(r ) 

steps} the «C-entropy of the scheme is if the 

outcome of the scheme X was the event x^. 

J 

So using (62) of chapter 2 we shall have 


(r+i) 

>/ 


H 4- 

i(«<-3/k:)k' 


S Pil 

4 


H 


(r) 


4‘< 


( 8 ) 


Multiplying 8) by p^^ , summing over i we shall have 

(rtl) (1) (r) 

^ H 4- 2 H . ( S p. p. .) 

^ (oc-l/k)}!' 4 i ^ ^4 


(9) 


Using (4) and (8)» (9) reduces to the required result given 
by (7), 

(ii) The result is trivially true for r « 1 i.e. 
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Suppose it is true for r = m i.e. 

(m) (1) 

^ HI max 

If ve argue as in result (i), take X to be the 
scheme describing m steps folloving Ai» X describing 
the m+l th step following the result of X and x be the 
outcome of X then by (69) of chapter 2 iwe have 

(ffi^l) (m) (l) 

S ^ max (11) 

Multiplying (11) by p^ , summing over i, changing the suffix 
i to i and Using (lO) we shall have 

(m+l) (1) 

^ (m+l) max 

whence the required result follows; 

Similarly following the above procedure and using 
(70) and (74) of chapter 2 we can prove results (iii) and 

(iv). 

Q.E.!). 

g If the transition probability matrix [Pfj] 

*,,jn of a Markov chain is in. the for® of a channel matrix 
with symmetric noise structure then the different steps 
«« entropies of the chain are additive i,e. 



(r) 


EI20| jto [Pij] Is of order 1 . ® 0 for all r 

jPi^] Is Qt » 

Let 


p s 


V' 


% 


1 

P 2 


A 2 

Po” 


with p^+ P2 = 1 


^ One step a state kj^ can hecome 

'^l prob, p^ 

-• Ag prob. pg 

and thU-s proceeding as in (l) 

Cl} at ^ V 

^ » (3/l-<) log Cp^^ + P 2 ^ «(> 0, ^ 1 


In two steps a 

state can become 


2 


with prob. p;j^ 


with prob. p^ pg 

Ag -• Ag 

Mlth prob. Pg 

2 

Ai Ag ^ Ai 

With prob. pg 


But these probabilities are the various teims of (Pi+P2>' 


( 2 ) . o( « .2 

» 


( 1 ) 


log (p, + % ) * 2 H 


hence 
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Similarly in x steps the state can become 
Ai - Ai Ai with prob. p^ 

r-1 

^1^ ^ prob, P2 

This second event can be had in ( ^ ) ways, each with 

prob, p^ Pg, since A^ can go to A^ in any r-l steps 

out of the totail number of r steps to be tak©i by it« 

Similarly probabilild.es of other events can be had. All 

these probabilities are nothing but the various terms in 
T 

(P]+ ^2^ • ®^®^ce 

(r) a( o( A (1) 

\ * (3/l-<) log (p^ + Pg ) * r 0,oc 4 1,00 

t^tocn is of order 3 

A^ &2 ^2 

\ Pi Pg P3 

bet P * A2 P2 P3 ^th p^+ P2+ P3 » 1 

^ P3 Pi ^2^ 

(1) oC «< or 

eQ » (1/1^) log P2 P3 ^ 

In two steps the state A^^ can become 

9/ 

Ai with prob. pj 
A^- A^’* Ag with prob. p^pg 

h with prob. p^pg 
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^1** h'* 

with proh. 

^2 

Ai** Ag-* Ag 

with prob. 

H P3 

Ai- Ag-^ Ag 

with prob. 

^2 Pi 


with prob. 

2 

% 

Ai- Ag- Ag 

with prob. 

P3 Pi 

*1* *3-* As 

with prob. 

P3 Pg 


Since these probabilities are the various terms of (P2^+P2+P3)^> 


so 


( 2 ) 


H 


0( 0 ( 2 (l) 

log Cp^ + Pg + pg ) * 2 


«C> 0, 

«{ 4 1 


similarly for r steps ve shall have 


(r) 


(l) 
r \ 


«< > 0 , « 4 1 . 


for the various probalilities for r steps from any state to 
start with can be had as the various terms of (pj+ pg^ pg) . 

^n is of order n, proceeding in the 

same manner as in (1) ve shall have 


oL at 

* (l/l-^) log (p^ +•....+ Pjj ) < ^ 0,aC 4 1 

and arguing the same way as above we shall have for this case 

Cr) fl{ o( r (l) 

H. » (l/l“<<) log (Pn * •••*+ Pn ^ t Of <<4 1 (13) 

o( X 
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In the limit, as 1, (13) tends to the known 

C 2*^ C ^ 

result * r ^ , Hence comhlnlng this symlx>llcally 

with (22) we shall have (12). 

Q. £• D. 

Remark Shannon in his definitton of entropy 

^(X) * • I p(x^) log p(Xj^) 

covered the case, when some of the p(x^) are zero, by laying 
doi» the convention that 0 xoo* 0. This is otherwise true 
in the extended res^ nmber system. In the same fashion if 
we adopt the same convention or consider the evaluations of 
our “^-entropies in the extended real number systen, Eenyi' s 
oC-entropy lidien some p(Xj^) « 0 and hence our =<-entropies given 
by (2) and (3) would be valid even if the- some of the p^^ 
are zero. 


PART - II 

1. OC.-EKTR3PY M b US EnST£KG£ 

In section 1.1 “(-entropy associated with a stationary 

source is defined and in section 1.2 its existence is established 
for a source wlmn the channel through which its contents are to 
be transmitted has a symmetric noise structure. 

1.1 oC-Batropy If 

Consider the set A * , x^^, ....IJ 

of all inUnite on toth sides, constructed from a s.t 

i Of the letters of a givod .lP‘^'«>t. Ib. subset of all those 
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I 

^ -q v-iice^ of A everyone of whicb. has specified letters 
certain ^ecified positions is fenowi as a set or briefly 
a cylinder. If te the smallest Borel field containing 
all such cylinders then ve know that a probability measure 
determining the probabilities of all such cylinders will 
uniquely determine the probability for every member of 
Let /* be such a probability measure. Further a sequence 
X * Xq, x^i.,.) is called stationary if its 

probability remains invariant under a shift operator X (say) 
i.e. if /-(T x) a ^ (x) , 

So, following Khinchin (1957), a discrete stationary 
source is defined as a>'4©uH.y-~sta^to»asy~i50urce~d>S‘'d«f3»ed-^^ 
a doubly infinite stationary sequence of randcm variables 

J[^, X^, ..... taking values on a finite set h, called the 
alphabet of the source. 

Suppose A has a letters and consider all n tejsns 

sequences j they will obviously be a*^ in nmbep. Every such 

sequence C(say) of the whole lot of a° in number is a 

cylinder set in the space A^ consisting of all x 6 A^ for 

which the z......* x.. , assime the fixed values characterizing 

t' t^iv* 1 

the given sequence C, a»d therefore has a definite probability 
/^(C). So the set of all such n-teim sequences repres«ats 
a finite probability space of a’^ elementary events C with 
probabilities /'(C). Hence following the procedure adopted for 
getting 

» (i/i*^) 2 r(c) 0, «c 4 1 (1) 

associated with such a space. 



If the soTirce emitting these n-term seqtiences 

is stationary then ^ in (i) «ill he Independent of 

the moment source started vorking and depends only on 

n and the nature of the source* So the average amount of 

“C-entropy accrued per symbol from such a source would he 
n 

/n. Since n is arbitrary and there is no limit to 

its choice, so different values of n will give rise to 

n 

different values of i|^ /n and thus we shall get a sequence 
n 

of /n. If this sequ^ce converges, it would he natural 
to call the limit, to which it converges, as the °<-entropy 
per symbol of this source* Denoting it by we have 

n 

“ I»t %r /n , «< > 0, *< I 1 (2) 

^ n 

the following section will prove that exists 
for a stationary source when the conditional probability 
matrix obtained from (^mbinlng two stationary sources 
consist^ of identical rows irrespective of the arrang^ent 
of their elements* 

2.2 

It is given in the fozm of the folloi4.ng theormmt 

n 

Th* 1 Establish the convergence of i|j /n for a 
stationary source identified with a stationary symmetric 
chame* 

Proof . The space S^^^of sequences of lei^th Br+n can be 
regarded as the product of the space of sequences of 
length m and the space ^ of sequences of length n. 
Hsnce using (47) and (76) of chapter 2, we shall have 
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^ ^ BL( (%) “C ^ 0, o( 1 

In tile present notation tills can be written as 
m+n 

Using (37) of chapter Si; 

n m+n 


(3) 


(4) 


putting m » 1 , (4) reduces to 


H. 


n+1 

4 ^ 


( 6 ) 


and putting m * n, (3) reduces to 

^ 2 ^ ( 6 ) 

By repetitive use of (6), (6) can be generalised to 
nk k 

^ ^ n (7) 

Putting k a 1 , (7) reduces to 

^ a 4 

or ^ /n ^ ^ < oo (8) 

(8) and the fact that /n^ 0 establish that /n 
is bounded above ^d below so by Boltzano-lifisijstrauss theorem 
the seq, / n j has both upper and lover limit. Let 


n 


Lt inf* H, / n 


00 


a « 
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SO to prove the ctnvergence of /n] it is sufficient 

to show that Lt ]^n « por an £ > 0 , we 

n •*oo ^ 

can choose an integer r such that 

1 /r - a j < e 
In particular this can he written as 

^ / r < a + e (9) 

For an n we can select an integer k > 0 such that 

(k-l) r < n ^ k r (10) 

From (lO) and (6) we have 
n kr 

This combined with (7) would give 

Q wi 

JJI j* 

Or /n ^ kr /n 1 ^/t (11) 

Using (9) and (10), (11) reduces to 

^j(/n < (k/k-l)(a ♦£) (12) 

Hmce for sufficiently large n (and hence for appropriately 
large k) (18) reduces to 

a«6 <^/n <a+2e 

and from this for arbitrarily small £ we shall have 
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Lt Sup. ^ /n s 


a 


n -*00 


Q* E*D. 


2 . 

Consider again the transmission through a channel 

of the members of A?'’ = ^(....» Xq, Xj^,...*)] • To 

every transmitted sequence x there corresponds a received 

I 

doubly infinite sequence belonging to the set B = 

. . . . , y. 3 _»yo» yi» • • • • )] constructed from the output 
alphabet B. The probability that in the received sequence 
Yn = b may be regarded in general- to depend on the whole 
sequence x and is the conditional probability that the 
sequence y 6 of received signals will belong to cylinder 
= b. But to characterise the channel completely we must 
toow not only all such probabilities but also of more complex 
sets S e Fg (smallest Borel field containing ET). 

Hence following Khinchin (1967) a channel is 
completely specified If we too« Its (1) Input alphabet A 
(11) output alphabet B and (UD tbe set of all probabilities 

■fUS) for any X e a' and any S e Fg and It Is symbolized 

by R, -crx> b] 5 

and the soured slv- rise to a new source [o.^ 

termed as the compound source where 0 =AxB, C = AxB 

and wCS) > for every S B Fg, la given hy 


„( 3 ) = w(H X N) = ; 


(la) 


putting M = a«a H 6 Fg arbitrary, ve shall have 

„(H X B) as t.m probability of the Joint occurrence of 
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X £ A and y $ H j tjut the former is the source so wCH x N) 
may be taken e<iual to (H) as the probability of occurrence 

of N £ Pg at the channel output . Thus IfCN) plays the 
same role for the space b" as /^(M) does for the space A^ . 
Hence (13) reduces to 

'^1 (N) as w(A^ X H) as f ^^(N) d (x) 


and this gives rise to the source at the channel output. 

We also know from ICkcvvcIkA^: (i<=«5'7} : that if ^t)^] and|A>^x,]^ 
are stationary then ^ and are also stationary. 

In section 1 we have established that every stationary 
source feeding a jsymmetric channel has a definite ®(-entropy. 

So if is effected by symmetric noise then |a, fj] 

would have definite “t-entropy attached to it. Following the 
same procedure as used for establishing the existence of 
Lt /n we can prove that there would be definite ®<“entropy 

1*1 wftOO 

attached to p,>l] . the validity of (3) for symmetric channels 

would make the oC-entropy associated with Jc, ^ exist. 

Sfe also know that specifying the *’pair sequence** 

Xq, y^, y^-i Is equivalent to specifying the pair 

of sequences x^iX^t %-i sn^ ^n-l » 

space of sequences **••» ^n-l> ^n-l product of 

the spaces of sequences Xo»****» %-l ^o» yi**‘*» yn-1* 

In other words the scheme constituting the elementary events of 

^ «. w -r 1 . v« n is the outcOTe of a ran<tom 

the type Xj^,yo,*..*» %-li yn-1 as “ 
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associatloii of the scheme constituting elementary events of 
the type ^o»****»^-l scheme constituting elemen- 
tary events of tye type Hence following the 

procedure adopted for deriving (47) of Chapter 2, we can 
establish that 

(Y/X) ^ ^ (Y) 1 (14) 

suad similarly 

n n 

^(X/Y) 4 : (X) «c^ 0, ^ 1 (15) 

From (14) follows the esdstence of 

IL(Y/X) 

Lt JZ * ^<(Y/X) (say) 

a n 

0 , << 1 . 

and from (16) follows the existence of 

n 

^ (X/Y) 

Lt .2 « ^(X/Y) (say) 0, 

OQ n 

^ ^ 1 . 

ao based on (l4) and (15) there wotild be two types 
q£ «(*-informations conveyed about X by Y and information 
about Y by X l.e. 

iS (XiX) * vn - 

add 


°‘7 


( 16 ) 
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(Yi Xi n (Y) - E^(Y/X) 0, ^ 1 


(17) 


Dividing (16) and (17) by n and taking limitsi as n -* oo , 
we shall kave o(-inforiaations per symbol of type I and 
type II. Denoting thea by I,^(X; Y) and I^CY; X) we have 

\iX) ^ iX/Y) 
n n 


I^ (X;Y) * Lt 

^ 00 




X} ^ LIS' 

XI 00 


B 

^(Y) 


(Y/X) 


n 


I (Xj Y) and I^(Y} X) may also be termed as 
rates per symbol (or signal) of «t-informatlons as 
conveyed about X by Y and as conveyed about Y by X . 


i 
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